Subject: [Roll] Toerless review: Retry outside outlook

From: Pascal Thubert <[email-masked]>
Date: Wed, 22 Sep 2021 10:22:18 +0200

Hello Toerless:

Many thanks for your time and care in this review. It was huge, involved reorganizing the doc and I published 20 to settle a base line:
https://www.ietf.org/archive/id/draft-ietf-roll-dao-projection-20.html

Alvaro, can you please have a look at Toerless' argument about update vs. extend around line 486?

Let's see:

> Thanks a lot for this work.  This is cool technology, it has the
> ability to provide very flexible and wide range of path steering
> and header optimizations for RPL networks.
> 

Sure! A note that your review comments suggest important reorgs (not redesign!) in the doc, which are mostly applied as suggested.
This will make diff review very hard going from 19 to 20. I hope things get stable after that. The diff is here:
https://www.ietf.org/rfcdiff?url2=draft-ietf-roll-dao-projection-20

OTOH, As the review progressed, I had to change and again, so proposed text below may have been updated or moved afterwards.
So I guess the best will be for you to skim through 20 to assess the changes. On the bright side, I added text to support ACP direct paths using P-DAO. 

Win-win 😉

> Overview:
> 
> I am not sure i understand all finer details due likely
> to some impedance mismatch between the expected RPL expertise of the text
> and mine, but i think that the proposed technology is sound
> and has on the technoloy (not text) side only few details that may
> need to be added (missing guidance or references on timers,
> maybe MTU handling if that is not supposed to be well-known, etc.).

All good points. For timers, RPL generally avoids absolute guidance though relative values of times may be indicated if there's a dependency. Because use cases vary so much.
For the MTU, I agree, let me add

                    In order to avoid fragmentation, e.g., using
  [RFC4944], [RFC8930], and/or [RFC8931] in a 6LoWPAN LLN, it is
  RECOMMENDED to allow an MTU that is larger than 1280 in the main
  DODAG and allows for the additional headers while exposing only 1280
  to the 6LoWPAN Nodes as indicated by section 4 of [RFC4944].

> 
> If there is any technological gap, then it is me hoping that this
> could be a solution for the RFC8994 A.9.4 problem, but alas this
> solution does not support establishment of additional routes for
> storage mode DODAGs. Too bad. Especially because its not quite clear
> to me why that would not be possible to easily do as well. But
> in understand how that RPL profile may not have been of any interest
> for this work.
> 

The only reason storing mode is excluded in this spec is simplification.
Because in that mode the Root is missing topology information. 
But that could be alleviated by indicating the parents as siblings.
I made a separate proposal based on your thread with Michael on ANIMA's need.
The proposal is a new section 10 that explains why the spec is designed for non storing in LLNs and how to use it in storing-mode non-LLNs.

> As i am not an expert for RPL, but mostly familia with only
> basic use of RPL, i am not sure how much of an expert a reader of this
> document is expected to be to be able to read, understand and vet this
> text.
> I think i and other RPL beginners would be able to understand the text
> and vet potential gotches better if more explanations where given,
> and my feedback indicates this accordingly. But i will not claim
> whether such explanations are always deemed necessary for the target
> audience.

Just what we need!

> 
> The mayor issue in the text is that it front-loads a lot of details
> that are then hard to understand and open questions are only answered
> much later. For example, packet encoding is put first in the document,
> as opposed to (as i have observed it in more RFCs) at the end. Then
> the text follows with processing description, and only finally with
> what i consider to be a complete description of how the pieces
> interact through examples. As mentioned in the details, i think the order
> should
> be as much as possible top-down, starting with sufficienly complex
> examples that introduce the technology to the extend that the
> main concepts are used in the example. E.g.: sequential or complex track,
> sements, destinations, targets.

OK, makes sense. I suggest to improve the terminology section to provide details on the main concepts as below:

2.3.  Main Concepts

  RPL is primarily designed to minimize the control plane activity,
  that is the relative amount of routing protocol exchanges vs. data
  traffic; this approach is beneficial for situations where the power
  and bandwidth are scarce (e.g., an IoT LLN where RPL is typically
  used today), but also in situations of high relative mobility between
  the nodes in the network (aka swarming, e.g., within a variable set
  of vehicles with a similar global motion, or a toon of drones), and
  in situations where the traffic is mostly directed from or to a
  central node, such as the control traffic between routers and a
  controller of a Software Defined Networking (SDN) infrastructure.

  To reduce the routing exchanges, RPL leverages an anisotropic
  Distance Vector approach, which does not need a global knowledge of
  the topology, and only optimizes the routes to and from the root,
  allowing P2P paths to be stretched.  Although RPL installs its routes
  proactively, it only maintains them lazily, in reaction to actual
  traffic, or as a slow background activity.  Additionally, RPL
  leverages the concept of objective function which allows to adapt the
  activity of the routing protocol to the use case, e.g., type, speed,
  and quality of the LLN links.  RPL does not need converge, and
  provides connectivity to most nodes most of the time.  The default
  route towards the Root is maintained aggressively and may change
  while a packet progresses without causing loops, so the packet will
  still reach the root.

  RPL first forms a default route in each node towards the a Root, and
  those routes together coalesce as a Directed Acyclic Graph upwards.
  RPL then constructs routes to so-called Targets in the reverse
  direction, down the same DODAG.

  In Non-Storing Mode, each node advertises itself as a target directly
  to the Root, indicating the parents that may be used to reach self.
  Recursively, the Root builds and maintains an image of the whole
  DODAG in memory, and leverages that abstraction to compute source
  route paths for the packets to their destinations down the DODAG.
  When a node changes its point(s) of attachment to the DODAG, it takes
  single unicast packet to the Root along the default route to update
  it, and the connectivity is restored immediately; this mode is
  preferable for use cases where internet connectivity is dominant, or
  when, like here, the Root controls the network activity in the nodes.

  In Storing Mode, the routing information percolates upwards, and each
  node maintains the routes to the subDAG of its descendants down the
  DODAG.  The maintenance is lazy, either reactive upon traffic or as a
  slow background process.  Packets flow via the common parent and the
  routing stretch is reduced vs. Non-Storing, for a better P2P
  connectivity, while the internet connectivity is restored more
  slowly, time for the DV operation to operate hop-by-hop.

  Either way, the stretched routing is not adapted to use cases that
  require high availability and reliability, such as; e.g., listed in
  RAW use cases [USE-CASES].  Extra stretch is counter-productive to
  both reliability and latency, and there's a need to better control
  the use of the available diversity, be it in time, frequency, path or
  technology.

  To complement RPL and eliminate routing stretch, this specification
  introduces an hybrid mode for RPL, that combines both Storing and
  Non-Storing modes to build Tracks from Ingress to Egress inside the
  DODAG.  A Track is typically a collection of parallel loose source
  routed sequences of nodes from Ingress to Egress, forming so-called
  Track Legs, and joined with strict Segments.  The Legs are expressed
  in RPL Non-Storing Modes and require an encapsulation to add a Source
  Route Header, whereas the Segments are expressed in Storing Mode.  A
  Serial Track comprises a single Leg and provides only one path
  between Ingress and Egress.  A Projected Route signals either a Track
  Leg or a Segment, using extended RPL DAO messages, and a Track is
  installed and maintained with a collection of the so-called Projected
  DAOs.

  But to provide redundancy, a Track must be more complex, either as a
  collection of Legs that may be parallel or cross at certain points,
  or as a more generic DODAG that is rooted at the Ingress Node.  This
  specification introduces the Via Information Option to signal a
  sequence of hops in a Leg or a Segment.

          CPF               CPF          CPF                 CPF

                         Southbound API

     _-._-._-._-._-._-._-._-._-._-._-._-._-._-._-._-._-._-._-._-._-._-
   _-._-._-._-._-._-._-._-._-._-._-._-._-._-._-._-._-._-._-._-._-._-

                        +----------+
                        | RPL Root |
                        +----------+
                          (      )
                (                                  )
          (              DODAG                              )
            (                                           )
    (                                                         )
                                                                    )
         Leg 1              B,                              E
    <-- Segment 1 A,B ---> <------ Segment 2 C,D,E ------>

               FWD  --z  Relay --z   FWD  --z   FWD          Target 1
           z-- Node  z--  Node  z--  Node  z--  Node --z     /
        --z    (A)        (B) \      (C)        (D)  z--    /
  Track                        \                       Track
  Ingress                    Segment 5                 Egress - Tgt 2
    (I)                           \                     (E)
        --z                        \                 z--    \
         z-- FWD   --z  FWD  --z  Relay --z  FWD  --z        \
             Node   z-- Node  z-- Node  z-- Node             Target 3
             (F)        (G)       (H)       (J)

    <------ Segment 3 F,G,H ------> <--- Segment 4 J,E --->
            Leg 2                  H,                      E

    <--- Segment 1 A,B ---> <- S5-> <--- Segment 4 J,E --->
            Leg 3          B,      H,                      E
                                                                    )
     (
                (                                        )

                      Figure 1: Segments and Tracks

  DetNet and RAW leverage a Controller Plane Function (CPF) called the
  Path Computation Element (PCE) [RFC4655] that interacts with RAW
  Nodes over a Southbound API to set up Tracks.  With this
  specification, the PCE interacts with the Root and the Root
  translates that signaling into RPL signaling.  The southbound
  interface is out of scope.  Forwarding Nodes only understand the
  simple 1-to-1 forwarding sublayer transport operation along a segment
  whereas the more sophisticated Relay nodes can also provide service
  sublayer functions such as Replication and Elimination.  One possible
  mapping with this specification is to signal the Relay Nodes as the
  hops of a Leg and the forwarding Nodes as the hops in a Segment that
  joins the Relays as illustrated in Figure 1.

  Note that while this specification enables to build both Segments
  inside a Leg (aka East-West), such as Segment 2 above which is within
  Leg 1, and Inter-Leg Segments (aka North-South), such as Segment 2
  above which joins Leg 1 and Leg 2, this specification does not signal
  to the Ingress which Inter-Leg Segments are available, so the use of
  North-South Segments and associated PAREO functions will require
  additional specification.  With this work, the only possibility is to
  define overlapping Legs as illustrated in Figure 1, with Leg 3 that
  is congruent with Leg 1 till node B and congruent with Leg 2 from
  node H on, abstracting Segment 5 as an East-West Segment.

> I only performed very opportunistic spell check, please do one yourself.
> Please run idnits check too, it complains about the abstract for me.
> 
> Appended draft with numbered lines and non-numbered line feedback/comments.
> 
> Cheers & thanks again
>    Toerless
> 
> 
>     5    ROLL                                                     P.
> Thubert, Ed.
>     6    Internet-Draft
> Cisco Systems
>     7    Updates: 6554 (if approved)
> R.A. Jadhav
>     8    Intended status: Standards Track
> Huawei Tech
>     9    Expires: 28 January 2022                                    M.
> Gillmore
>    10
> Itron
>    11                                                                27
> July 2021
>    12
>    13
>    14                      Root initiated routing state in RPL
>    15                       draft-ietf-roll-dao-projection-19
> ...
> 
>   135    1.  Introduction
>   136
>   137       RPL, the "Routing Protocol for Low Power and Lossy Networks"
> [RPL]
>   138       (LLNs), is a generic Distance Vector protocol that is well
> suited for
>   139       application in a variety of low energy Internet of Things
> (IoT)
>   140       networks.  RPL forms Destination Oriented Directed Acyclic
> Graphs
>   141       (DODAGs) in which the Root often acts as the Border Router
> to connect
>   142       the RPL domain to the Internet.  The Root is responsible to
> select
>                          ^^^^^^^^^^^^^^^
> Really ? What is the "often" case where e.g.: at least nodes in the RPL
> network
> can reach nodes across the Internet ? Aka: from my understanding, the
> common
> deployment would attach to other limited domain (RFC8799) networks, for
> example
> if i correctly glean from rfc9030, but not "the Internet". Otherwise,
> please
> provide a reference to justify "often ... to the Internet".
> 
> Aka: suggest to replace Internet with "other network" or maybe "a
> backbone".

Makes sense. A backbone is fine.

> 
>   142       the RPL domain to the Internet.  The Root is responsible to
> select
>   143       the RPL Instance that is used to forward a packet coming
> from the
>   144       Internet into the RPL domain and set the related RPL
> information in
> 
> Same comments about "the Internet".

Ack

> 
>   147       The "6TiSCH Architecture" [6TiSCH-ARCHI] also leverages the
>   148       "Deterministic Networking Architecture" [RFC8655]
> centralized model
>   149       whereby the device resources and capabilities are exposed to
> an
>   150       external controller which installs routing states into the
> network
>   151       based on some objective functions that reside in that
> external
>   152       entity.  With DetNet and 6TiSCH, the component of the
> controller that
> 
> The DetNet architecture has a lot more aspects for which i am not sure,
> that
> 6TiSCH matches them (have not gone fully through 6TiSCH-ARCHI). Aka: DetNet
> as
> of today implies the per-hop,per-flow IntServ architecture, but no DiffServ
> for DetNet service hops. Is this what 6TiSCH-ARCHI does as well ?

The sentence above means to say that we leverage the centralized model, not necessarily apply all DetNet concepts at all times.
6TiSCH ARCHI did not describe a service layer per se though the functionality was assumed as multiple in / multiple out time slots, regardless of who is speaking or listening.
A 6TiSCH Track Segment MUST be strict as it matches DetNet forwarding sublayer; and it make sense in RAW cases to associate the source routed hops of a Track Leg to the (service layer) relays.
As it goes, 6TiSCH inherits from RAW which inherits from DetNet and yes the architecture would be the same in my mind. But not necessarily with the goal to enforcing deterministic properties (like bounded latency) all the time. The text I added for your earlier point in termminology covers that I hope.

> 
> If not, then maybe change to "levereges the aspects of the ..." to make
> the claim more scoped to what the paragraph describes.

Changed to 

  The "6TiSCH Architecture" [6TiSCH-ARCHI] leverages a centralized
  model similar to that of "Deterministic Networking Architecture"
  [RFC8655], whereby the device resources and capabilities are exposed
  to an external controller which installs routing states into the
  network based on its own objective functions that reside in that
  external entity.  With DetNet and 6TiSCH, the component of the
  controller that is responsible of computing routes is called a Path
  Computation Element ([PCE]), and the objective functions are
  described in [RFC4655] and encoded in the PCE Protocol (PCEP) by
  [RFC5551].

> 
>   152       entity.  With DetNet and 6TiSCH, the component of the
> controller that
>   153       is responsible of computing routes is called a Path
> Computation
>   154       Element ([PCE]).
>   155
>   156       Based on heuristics of usage, path length, and knowledge of
> device
>   157       capacity and available resources such as battery levels and
>   158       reservable buffers, the PCE with a global visibility on the
> system
>   159       can compute direct Peer to Peer (P2P) routes that are
> optimized for
>   160       the needs expressed by an objective function.  This document
> 
> "Based on heuristics" has one doubt whether the result is deerministic. Not
> saying heuristics may not play a role, but maybe fnd a better wording.

The idea is that the controller sees reliability stats and based on those stats it computes how many parallel paths are needed and how many copies of the packets need to be sent to meet the SLA.
The point about DetNet is not to make necessarily deterministic paths but to use a similar model with a controller with "God's view".
Now there can be AI in the controller and if so, it turns the statistics / time series in heuristics. Open to change but there may be loss of information.
So I guess that the language is correct, open to more discussion.

> 
> "global visibility" sounds like weather satellites. More specifically when
> reading, i wonder if the scope of the discussion in this document can be
> constraint to a single RPL network, or if it is important to understand
> that the mechanisms of this document a also enabling more "interesting"
> paths.

As Anand pointed out, the metrics that are reported to the controller and how that happens is out of scope. Added text on that for his review, will be part of the same diff set from v19 to 20.
But yes, the ultimate goal is as you say, basically TE / DetNet paths.

> 
> Aka: If the following is one of the cases for which this drafts work is
> in support of, then i would really like to see this type of picture and
> explanation added as something like a reference picture with explanation:
> 
>               Client11               Client21             ClientN1
>                |                      |                    |
>  PCE        RPL-network1         RPL-network2         RPL-networkN
>   |        Root1     Rtr1b      Root2    Rtr2b  ...   RootN   RtrNb
>   |          |         |          |        |           |       |
>  ..................................................................
>                       Backbone Network, L2 / L3, e.g.:
>  -+----------+---------+----------+--------+-----------+-------+----

What is rtr1B ? backup for Root? It seems connected to the RPL network; is it? if so, that's not in scope but we could add it to the thought process.

The global network may be as you say but the scope of this doc is within one DODAG.
Continuing a track across another DODAG is feasible but out of scope. Again t keep simple for now.
And again if you have a string use case, the group may change its mind.

> 
> Aka: The PCE can calculate paths between clients in this internetwork.
> An example path between Client11 and ClientN1 is composed of the
> projected path between Client11 and one of the nodes connecting RPL-
> network1
> to the backbone, e.g.: Rtr1b, a path through the backbone, and finally
> a path through RPL-networkN, again including the choice of edge
> router to the backbone, e.g.: RtrNb.

While feasible, this is a story way beyond our spec. For doing the above, we'd not build a Track, but use plain RPL as it is already the shortest path.
Though we use a controller the spec is only a first step towards RAW / DetNet for routing. All the metrics to the root, the policies to select which flow goes in which Track, the schedules or whatever, need to come from RAW/6TiSCH_2.0.
If you need something like this for ANIMA, let's chat outside this thread.

> 
> I guess if the solution does not support RtrIb, then its explanation
> could more easily be constrained to just a single RPL-network.

With RPL the Root is the only connection to the "backbone". That's why it is usually the same box as the 6LoWPAN 6LBR.

> 
>   160       the needs expressed by an objective function.  This document
> 
> This statement is confusing.
> Without this draft, the objective functions result in specific paths,
> and these paths are insufficient, so the PCE also uses this drafts
> mechanism to achieve better paths. So the objective function clearly
> was not sufficient. Besides, the objective function is very specific
> to parameters used in distributed path calculation, not PCE.

The PCE has its own objective functions, see RFC 4655, that are encoded in PCEP by RFC 5551. 
The goals of a Track are different from those oif the lain DIDAG so the OF will be different. Here we are talking about the Track OF as seen by the PCE. 
Let me add a reference. With the above discussion this becomes:

 The "6TiSCH Architecture" [6TiSCH-ARCHI] leverages a centralized
  model similar to that of "Deterministic Networking Architecture"
  [RFC8655], whereby the device resources and capabilities are exposed
  to an external controller which installs routing states into the
  network based on its own objective functions that reside in that
  external entity.  With DetNet and 6TiSCH, the component of the
  controller that is responsible of computing routes is called a Path
  Computation Element ([PCE]), and the objective functions are
  described in [RFC4655] and encoded in the PCE Protocol (PCEP) by
  [RFC5551].
> 
> In my mind, there is some goal/objective for the path to be calculated,
> and this objective is achieved by OF to create DODAG and paths, and if
> those paths are insufficient, then
> and a controller can achieve that objective by defining a
> but when the controller determinses that it can not solely be achieved
> through objective function parameters, then it starts using this drafts
> mechanism...

Well, think that in non-storing mode all packets are routed through the Root. Very inefficient.
So the function N1 of the draft is to do shortcuts with specific SLAs that will be expressed as PCE OF (not RPL OF though it's all very abstract anyway).
The function N2, of value only for non-storing, is to make the RH smaller for routes in the main DODAG, using storing mode Segments to join the loose SRH.

> 
>   160       the needs expressed by an objective function.  This document
> 
> Suggest new paragraph before "This"
> 
>   161       specifies protocol extensions to RPL [RPL] that enable the
> Root of a
>   162       main DODAG to install centrally-computed routes inside the
> DODAG on
>   163       behalf of a PCE.
>   172
>   173       This specification expects that the main RPL Instance is
> operated in
>   174       RPL Non-Storing Mode of Operation (MOP) to sustain the
> exchanges with
>   175       the Root.  In that Mode, the Root has enough information to
> build a
>                    ^ Only ?
>   176       basic DODAG topology based on parents and children, but
> lacks the
>                                                                  ^ still
>   177       knowledge of siblings.  This document adds the capability
> for nodes
> 
> Aka: in storing mode the root would even have less information, right ?

Exactly; in NS we have the DODAG at the root already. See my point above

> 
>   178       to advertise sibling information in order to improve the
> topological
>   179       awareness of the Root.
>   180
>   181       As opposed to the classical RPL operations where routes are
> injected
>   182       by the Target nodes, the protocol extensions enable the Root
> of a
>   183       DODAG to project the routes that are needed onto the nodes
> where they
>   184       should be installed.  This specification uses the term
> Projected
>   185       Route to refer to those routes.  Projected Routes can be
> used to
>   186       reduce the size of the source routing headers with loose
> source
>   187       routing operations down the main RPL DODAG.  Projected
> Routes can
>   188       also be used to build transversal routes for route
> optimization and
>   189       Traffic Engineering purposes, between nodes of the DODAG.
>   190
>   191       A Projected Route may be installed in either Storing and
> Non-Storing
>   192       Mode, potentially resulting in hybrid situations where the
> Mode of
>   193       the Projected Route is different from that of the main RPL
> Instance.
>   194       A Projected Route may be a stand-alone end-to-end path or a
> Segment
>   195       in a more complex forwarding graph called a Track.
>   197       The concept of a Track was introduced in the 6TiSCH
> architecture, as
>   198       a potentially complex path with redundant forwarding
> solutions along
>   199       the way.
> 
> Please add a reference to the spec that introduced Track.

Done, that's RFC 9030 

> 
> Where is "stand-alone" introduced ? If elswhere, add reference, otherwise
> say its introduced here. Also, everything else is capitalized, maybe you
> want
> to capitalize also Stand Alone.

Yes that term could be useful. Adding to terminology

> 
>   199     With this specification, a Track is a DODAG formed by a RPL
>   200       local Instance that is rooted at the Track Ingress.  If
> there is a
>   201       single Track Egress, then the Track is reversible to form
> another
>   202       DODAG by reversing the direction of each edge.  A node at
> the ingress
>   203       of more than one Segment in a Track may use one or more of
> these
>   204       Segments to forward a packet inside the Track.
> 
> The last sentence is using the term Segment to explain something new, but
> the
> term Segment was not sufficiently defined in before. Other than that a
> Segment
> can be or is always (?) a Projected Route, but Projected Route is also
> undefined.

This is now all introduced in the new Terminology section.

> 
>   206       The "Reliable and Available Wireless (RAW)
> Architecture/Framework"
>   207       [RAW-ARCHI] defines the Path Selection Engine (PSE) that
> adapts the
>   208       use of the path redundancy within a Track to defeat the
> diverse
>   209       causes of packet loss.
>   210
>   211       The PSE is a dataplane extension of the PCE; it controls the
>   212       forwarding operation of the packets within a Track, using
> Packet ARQ,
>   213       Replication, Elimination, and Overhearing (PAREO) functions
> over the
>   214       Track segments, to provide a dynamic balance between the
> reliability
>   215       and availability requirements of the flows and the need to
> conserve
>   216       energy and spectrum.
>   217
>   218       The time scale at which the PCE (re)computes the Track can
> be long,
>   219       using long-term statistical metrics to perform global
> optimizations
>   220       at the scale of the whole network.  Conversely, the PSE
> makes
>   229       forwarding decisions at the time scale of one or a small
> collection
>   230       of packets, based on a knowledge that is limited in scope to
> the
>   231       Track itself, so it can be refreshed at a fast pace.
> 
> I would suggest to restructure this. Before going into the RPL details as
> you do
> in before, maybe start with the problem to be solved. Which seems to be a
> mechanism to support packet replication, traversal over alternative paths
> and
> duplicate elimination (and seemingly more). 6TiSCH introduced Tracks as the
> concept for redundant paths, RAW inroduced PAREO. This document introduces
> one solution to to build Tracks with RPL und a mechanism called Projected
> Routes.
> 

Agreed, did that in the new Terminology section too. Note that PAREO is not signaled in this spec, new text on that too.

> Something like that to have a more logical sequence of explanations, top to
> bottom.
> 

Ack

>   233       Projected Routes must be used with the parsimony to limit
> the amount
> 
> Parsimony in he sense you use it, was never used in RFCs, rc871 used law of
> parsimony.
> Suggest to reword for easier reading. For example, "Care must be taken that
> state installed for
> projected routes fit in each devices resources".

What about

  Projected Routes require resources in the routers; the amount of
  state that is installed in each node must be computed to fit within
  the node's memory, and the amount of rerouted traffic must fit within
  the capabilities of the transmission links.  The methods used to
  learn the node capabilities and the resources that are available in
  the devices and in the network are out of scope for this document.
  The method to report the LLN link capacity and reliability statistics
  are also out of scope.  They may be fetched from the nodes through
  network management functions or other forms of telemetry such as OAM.

> 
> Is the amount of traffic solely a responsibility of the installation of
> projected
> routes ? Isn't this also responsibility of the PSE ? (just guessing,
> haven't read
> up on it).

You're quite right, but all that discussion is out of scope; as far as this 
Doc is concerned, there's no ingress policy and not associated PREOF intent.
All the spec can expect in implicitly classical ECMP over redundant paths.
Text added in the introduction covers that, see a little below in this review.

> 
>   234       of state that is installed in each device to fit within the
> device
>   235       resources, and to maintain the amount of rerouted traffic
> within the
>   237       capabilities of the transmission links.  The methods used to
> learn
>   237       the node capabilities and the resources that are available
> in the
>   238       devices and in the network are out of scope for this
> document.
>   239
>   240       This specification uses the RPL Root as a proxy to the PCE.
> The PCE
>   241       may be collocated with the Root, or may reside in an
> external
>   242       Controller.
>   243
>   244       In that case, the PCE exchanges control messages with the
> Root over a
>              ^^^^ the latter
>   245       Southbound API that is out of scope for this specification.
> The
>                     ^ PCE
>   246       algorithm to compute the paths and the protocol used by an
> external
>   247       PCE to obtain the topology of the network from the Root are
> also out
>   248       of scope.
> 
> In his AD review for draft-ietf-bier-te-arch, Alvaro asked me to provide
> examples
> for such topology discovery. YOu can of course wait out if this will happen
> for
> his doc too, i personally have no strong opinions, but if you need example
> text, see draft-ietf-bier-te-arch, 3.2.1.1, Third paragraph.

Well, that could be the same RPL formats, or anything else. 

The section would become:

  There specification considers that there's a unique PCE with full
  view of the network.  Distributing the function for a large network
  is out of scope.  This specification uses the RPL Root as a proxy to
  the PCE.  The PCE may be collocated with the Root, or may reside in
  an external Controller.  In that case, the protocol between the Root
  and the PCE is out of scope and abstracted by / mapped to RPL inside
  the DODAG; one possibility is for the Root to transmit the RPL DAOs
  with the SIOs that detail the parent/child and sibling information.

> 
> Personally, i do not like the conditioning that a southbound API is
> supposedly
> only necessary when the PCE is external from the RPL Root. IMHO, there
> always
> needs to be at least an informat model for Projected Routes. Which you may
> agree on, but its not mentioned. PCE and RPL Root can just be independent
> software entities on the same node.

Very true; we'll need a spec for that

> 
> 
>   260    2.2.  Glossary
>   261
>   262       This document often uses the following acronyms:
>   263
>   264       CMO:  Control Message Option
>   265       DAO:  Destination Advertisement Object
>   266       DAG:  Directed Acyclic Graph
>   267       DODAG:  Destination-Oriented Directed Acyclic Graph; A DAG
> with only
>   268          one vertex (i.e., node) that has no outgoing edge (i.e.,
> link)
>   269       LLN:  Low-Power and Lossy Network
>   270       MOP:  RPL Mode of Operation
>   271       P-DAO:  Projected DAO
>   272       P-Route:  Projected Route
>   273       PDR:  P-DAO Request
>   274       RAN:  RPL-Aware Node (either a RPL Router or a RPL-Aware
> Leaf)
>   275       RAL:  RPL-Aware Leaf
>   276       RH:  Routing Header
>   285       RPI:  RPL Packet Information
>   286       RTO:  RPL Target Option
>   287       RUL:  RPL-Unaware Leaf
>   288       SIO:  RPL Sibling Information Option
>   289       SR-VIO:  A Source-Routed Via Information Option, used in
> Non-Storing-
>   290          Mode P-DAO messages.
>   291       TIO:  RPL Transit Information Option
>   292       SF-VIO:  A Via Information Option, used in Storing-Mode P-
> DAO
>   293          messages.
>   294       VIO:  A Via Information Option; it can be a SF-VIO or an SR-
> VIO.
> 
> I had a hard time remembering SR-VIO and SF-VIO throughout the text. Maybe
> SM-VIO and NSM-VIO would be easier to remember and better because they
> explicitly
> refer to the explanation (Storing Mode, Non-Storing Mode).

Adopted

> 
> I'll spare you the request for a full Terminology section for these terms
> with
> references, maybe others will do so.

Did it

> 
> It would make sense though to move the terms newly introduced in this doc,
> such as P-Route , P-DAO and the like into section 2.3 and rename 2.3 to
> "New terms".

They are not necessarily new; a lot is copied from 6TiSCH or RAW.

> Derived terms like P-Route can of course stay concise.
> 
>   296    2.3.  Other Terms
> ...
> 
>   327    3.  Extending RFC 6550
> 
> You have tagged this draft as an Update to RFC6550. You should IMHO have a
> section where you summarize what that exactly entails, ideally call
> "Updated to RFC6550".
> 
> AFAIK, we do not have a good formal structure for such update description,
> Mirja
> tried to propose some more keywords, but so far, Update can mean anything.
> 

From the latest understanding I gather that we are extending but not updating, no impact on existing code.
So I removed the tag. Alvaro will help here I guess.

> I would suggest to summarize what type of update this is by making
> statements
> answering the following type of questions (maybe i forgot some). These
> statements
> are also useful when they are answering the questions below with No. I
> leave it up to
> you if you feel this would be good to go here in the beginnin, or if you'd
> rather
> want to summarize it in the end of the document.

We have 

  3.  Extending RFC 6550  . . . . . . . . . . . . . . . . . . . . .  12
  4.  Extending RFC 6553  . . . . . . . . . . . . . . . . . . . . .  15
  5.  Extending RFC 8138

> Questions:

Very useful list!

> Does this document specify any functionality that is REQUIREDS or
> RECOMMENDED
> for implementations of RFC6550 ? If so, please detail them.  Else, this can
> be called
> a fully OPTIONAL update for RFC6550.

It is optional. This can operate in a mixed network.

> Will implementations supporting this RFC continue to be backward compatible
> with
> all prior RFCs, RFC6550 and related ? If not, please describe any changes
> in
> interoperability.

They will

> 
> Does this RFC introduce any functional extensions to RFC6550 or other RFC
> through mechanisms other than pre-defined extension points ?
> 
> Are any pre-defined extension points of RFC6550 or other RFC used, which
> are not
> already IANA registries ?  If so, please include references to the original
> RFC
> section where that extension point was described.
> 

No such thing

> (first time i am trying to formalize these type of question...)

We need to store that somewhere, very useful

> 
> IMHO outside the scope of documenting the "Update" flag is mixed
> deployment.
> If there is no text for that, would be good to add that somehwere. Update
> section
> wouldn't be a bad place.
> 

I removed the update tag

> 
>   341    3.1.  Projected DAO
> 
> At this point i am worried whether i am correctly guessing at the
> forwarding
> plane behavior of projected routes, because i felt it was presented in the
> Intro section mostly en-passant, but it is hard to follow the document with
> the correct explanation of how forwarding works.
> 
> What i figured from the Intro is that Projected Routes would allow the root
> to specify a loose path in the RH, even though only non-storing mode is
> used.
> Whenever a RPL node then sees a next-op in the RH it looks for a projected
> Route towards that next-hop and hmm... just forwards the packet to the
> first next hop on the projeted route ? Or it needs to change the routing
> header to prepend the projected route ?

Makes sense, all this discussion on PSE is really extraneous. I changed the intro to

  The "Reliable and Available Wireless (RAW) Architecture/Framework"
  [RAW-ARCHI] extends the definition of Track, as being composed to
  east-west directional segments and north-south bidirectional
  segments.  It also defines the Path Selection Engine (PSE) that
  adapts the use of the path redundancy within a Track to defeat the
  diverse causes of packet loss.

  This specification prepares for RAW by setting up the Tracks, but
  does not provide ingress policy that selects which packets are routed
  through which Track, and how the PSE may use the path redundancy
  within the Track.  As far as this specification is concerned, the
  main DODAG provides a default route via the Root, and the Tracks
  provide mode specific routes to the Track targets.  The use of the
  redundancy is limited to ECMP load balancing, and all the segments
  are east-west unidirectional only.

> 
> And i have absolutely no idea what interactions there are between PSE and
> Projected routes in the forwarding plane..

I hope the text above clarifies

> 
> Section 9 has some signalling examples. Maybe it would be good to create
> some introductory example that shows the relevant forwarding plane aspects.
> 
>   343       Section 6 of [RPL] introduces the RPL Control Message
> Options (CMO),
>   344       including the RPL Target Option (RTO) and Transit
> Information Option
>   345       (TIO), which can be placed in RPL messages such as the
> Destination
>   346       Advertisement Object (DAO).
> 
>   346       This specification extends the DAO
>   347       message with the Projected DAO (P-DAO); a P-DAO message
> signals a
>   348       Projected Route to one or more Targets using the new CMOs
> presented
>   349       therein.
> 
> This is confusing. If i am guessing right, i would suggest to write:

> 
> This specification defines a new CMO carrying a Projected Route. When this
> Projected Route CMO is included in a DAO message, this is called a
> Projected DAO (P-DAO).

Used the proposed text with a twist; many thanks!

  This specification defines a new CMO carrying a Projected Route.
  A DAO message that conveys the new CMO is called a Projected DAO (P-DAO).

> 
> I do not get the "signals _a_ Projected Route to one or _more_ Targets".
> How can
> a single P-DAO be signalled to more than one recipient ?    

Storing Mode P-DAO flows from egress to ingress, installing the segment on the way ala intserv / rsvp.

> 
> 
>   349     This specification enables to combine one or more Projected
>   350       Routes into a DODAG called a Track, that is traversed to
> reach the
>   351       Targets.
>   352
>   353       The Track is assimilated with the DODAG formed for a Local
> RPL
>   354       Instance.  The local RPLInstanceID of the Track is called
> the
>   355       TrackID, more in Section 7.2.  A P-DAO message for a Track
> signals
>   356       the TrackID in the RPLInstanceID field.  The Track Ingress
> is
>   357       signaled in the DODAGID field of the Projected DAO Base
> Object; that
>   358       field is elided in the case of the main RPL Instance.  The
> Track
>   359       Ingress is the Root of the Track, as shown in Figure 1.
> 
> How about the text here first finishes up the specification of the P-DAO
> CMO
> before explaining some, but likely not all details of how this stuff
> works together ?

This is now described in the terminology section, with illustration.

> 
>   360
>   361       This specification defines the new "Projected DAO" (P) flag.
> The 'P'
>   362       flag is encoded in bit position 2 (to be confirmed by IANA)
> of the
>   363       Flags field in the DAO Base Object.  The Root MUST set it to
> 1 in a
>   364       Projected DAO message.  Otherwise it MUST be set to 0.  It
> is set to
>   365       0 in legacy implementations as specified respectively in
> Sections
>   366       20.11 and 6.4 of [RPL]. .
>   367
>   368          0                   1                   2
> 3
>   369          0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8
> 9 0 1
>   370         +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-
> +-+-+-+
>   371         |    TrackID    |K|D|P|  Flags  |   Reserved    |
> DAOSequence   |
>   372         +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-
> +-+-+-+
>   373         |
> |
>   374         +
> +
>   375         |
> |
>   376         +               IPv6 Address of the Track Ingress
> +
>   377         |
> |
>   378         +
> +
>   379         |
> |
>   380         +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-
> +-+-+-+
>   381         |   Option(s)...
>   382         +-+-+-+-+-+-+-+-+
>   383
>   384                        Figure 1: Projected DAO Base Object
>   385
>   386       New fields:
>   387
>   388       TrackID:  In the case of a P-DAO, the RPLInstanceID field is
> called
>   397          TrackID.  This is a naming convenience but does not
> change the
>   398          semantics and format of the RPLInstanceID that is used as
> TrackID.
> 
> Maybe then stick to calling it RPLInstanceID in the ASCII and not calling
> it
> a new field (nitpicking).

I must stick the the Archi (RAW, 6TiSCH) ; There could be other forms of TrackIDs.
Changing "called TrackID" to "used as TrackID"

> 
>   400       P:  1-bit flag (position to be confirmed by IANA).
>   401
>   402          The 'P' flag is set to 1 by the Root to signal a
> Projected DAO,
>   403          and it is set to 0 otherwise.
>   404
>   405       In RPL Non-Storing Mode, the TIO and RTO are combined in a
> DAO
>   406       message to inform the DODAG Root of all the edges in the
> DODAG, which
>   407       are formed by the directed parent-child relationships.
> Options may
>   408       be factorized; multiple RTOs may be present to signal a
> collection of
>   409       children that can be reached via the parent(s) indicated in
> the
>   410       TIO(s) that follows the RTOs.  This specification
> generalizes the
>   411       case of a parent that can be used to reach a child with that
> of a
>   412       whole Track through which both children and siblings of the
> Track
>   413       Egress are reachable.
> 
> This again seems to be mixing larger logical behavior explanation into what
> probably should be just the spec for the new message encodings here. See
> how
> you can decouple the larger system explanations from the encoding spec.

True enough but that's 2 sentences that may help the reader there. 
Let me add a fwd ref to 7.3.x where the details are, as below:

  Like in a DAO message, the RTOs can be factorized in a P-DAO, but the
  Via Information Options cannot.  A P-DAO contains one or more RTOs
  that indicate the destinations that can be reached via the Track, and
  exactly one VIO that signals a sequence of nodes.  In Non-Storing
  Mode, the Root sends the P-DAO to the Track Ingress where the source-
  routing state is stored, whereas in Storing Mode, the P-DAO is sent
  to the Track Egress and forwarded along the Segment in the reverse
  direction, installing a Storing Mode state to the Track Egress at
  each hop, see Section 7.3.2 and Section 7.3.1 respectively.  In both
  cases the Track Ingress is the owner of the Track, and it generates
  the P-DAO-ACK when the installation is successful.

> 
>   415       New CMOs called the Via Information Options (VIO) are
> introduced for
>   416       use in P-DAO messages as a multihop alternative to the TIO.
> One VIO
> 
> a) Add forward pointer to section defining VIO
> 

> b I get the feeling as if the whole mentioning of TIO and RTO above should
> simply
> go in a separate chapter for RPL experts with a title like "Why are TIO and
> RTO
> not sufficient". Or "How to combine TIO/RTO with SF-VIO" etc.. but try to
> stick to topic at hand here. Feeble minded readers like myself can only
> balance
> a limited number of partially understood terms in the air at the same time.

Moved stuff around, the section is now simpler and focused on P DAO formats 

> 
>   417       is the Stateful VIO (SF-VIO); the SF-VIO installs Storing-
> Mode
>                                                            ^ a ?
>   418       Projected Route  along a strict segment.  The other is the
> Source-
>                          ^ s ?
>   419       Routed VIO (SR-VIO); the SR-VIO installs a Non-Storing-Mode
> Projected
>   420       Route at the Track Ingress, which uses that state to
> encapsulate a
>   421       packet with a Routing Header (RH) to the Track Egress.
> 
> Question... If i just use basic RPL with non-storing mode, i already
> need an RH. And that is let's say a strict path. So after the first hop of
> that
> RH, my next hop is actually a track ingres for which i have SR-VIO. Does
> this
> mean i need to do a full new IPv6 encap of the packet to insert the new
> RH for the track ? 

Added a fwd link to 7.6.  Forwarding Along a Track
I guess you later found the detailed answer in for your question in  9.1.1.  Stitched Segments
The RH for the main DODAG becomes loose, and SM-VIOs connect the dots. No reencaps unless you really like them.

> If so:
> 
> a) i'd say: "to encapsulate the packet into an IPv6 packe with a new RH to
> the track egress.

done

> b) What about MTU issues due to the encap ? Can not find word MTU in text.

Was true. 7.3 now has 

  A Non-Storing-Mode P-DAO signals a strict or loose sequence of nodes
  between the Track Ingress (excluded) and a Track Egress (included).
  It installs a source-routed path of a higher precedence within the
  Track indicated by the P-DAO Base Object, towards the Targets
  indicated in the Target Options.  The source-routed path requires a
  Source-Routing header which implies an encapsulation to add the SRH
  to an existing packet.  In order to avoid fragmentation, e.g., using
  [6LoWPAN], [RFC8930], and/or [RFC8931] in a 6LoWPAN LLN, it is
  RECOMMENDED to allow an MTU that is larger than 1280 in the main
  DODAG and allows for the additional headers while exposing only 1280
  to the 6LoWPAN Nodes as indicated by section 4 of [6LoWPAN].

> 
>   423       Like in a DAO message, the RTOs can be factorized in a P-
> DAO, but the
>   424       Via Information Options cannot.
> 
>   424       A P-DAO contains one or more RTOs
>                  ^ MUST/SHOULD/MAY ?
> 

This is said later, in uppercase as below; adding a fwd link

   A P-DAO message MUST contain exactly one VIO, which is either a SM-VIO or an
   NSM-VIO, and MUST follow one or more RTOs.

>   425       that indicate the destinations that can be reached via the
> Track, and
>   426       exactly one VIO that signals a sequence of nodes .  In Non-
> Storing
>                                                           ^ That form the
> track ?

Now

  A P-DAO contains one or more RTOs to indicate the target
  (destinations) that can be reached via the P-Route, and at most one
  VIO that signals the sequence of nodes to be followed, more in
  Section 7.  

> 
>       root ....... Track-Ingres ...(Track Midpoints)... Track-Egres .....
> Destination1
>                                                                     ...
>                                                                     ...
> DestinationN
> P-DAO has:
>  one RTO for each DestinationI
>  one VIO for the Track: Midpoints,Egres
> 
> Semantic: When packet reaches Track-Ingres, it checks whether destination
> or
> next-hop is one of the DestinationI, if so, then use the track. If SR-VIO,
> then encap IPv6/RH, if SF-VIO, then just forward to first hop on track,
> which will also have installed the right state for the track...

True, that's basic routing since the Track provides a more specific route, and SR VIO (now NSM VIO) signals a tunnel.

> 
> Something like this with picture would make the whole concept so much
> easier to understand (to me).

There's now one with 2 legs in the terminology. And then the whole section 9.

> 
>   427       Mode, the Root sends the P-DAO to the Track Ingress where
> the source-
>   428       routing state is stored.  In Storing Mode, the P-DAO is sent
> to the
>   429       Track Egress and forwarded along the Segment in the reverse
>   430       direction, installing a Storing Mode state to the Track
> Egress at
>   431       each hop.  In both cases the Track Ingress is the owner of
> the Track,
>   432       and it generates the P-DAO-ACK when the installation is
> successful.
> 
> How would the root know whether in storing node all the track midpoints
> did successfully install the necessary track state ? If not, hen it would
> be
> worth pointing out how storing mode may end up being less reliable ?

There's an ack sent by the Ingress if OK or any node along the segment if not OK. There's retries. But arguably there's more complexity and more chances to fail. 
I'd rather leave that for the reader to estimate in his case.

> 
>   434    3.2.  Sibling Information Option
> 
> Is there any reason why we wouldn't first want to have the VIO section here
> so a feeble minded reader could pop off some open definitions of the stack
> and declare understanding victory on whats explaind so far ?

Added, the text was there but the section missing

> 
> To answe my own question, i think your structure is around whatever RFC
> is supposedly extended by a new semantic introduced, but that results in
> terrible flow of reading. I would strongly suggest to  reshuffle the
> text from the most simple cases to the most complex cases and have
> separate short sections summarizing for each affected RFC how it is
> extended usin the template questions above, using references to the main
> specification part.

OK, I see you. I moved section 9 up so we see what we're trying to achieve before we learn how that is done

> 
>   436       This specification adds another CMO called the Sibling
> Information
>   437       Option (SIO) that is used by a RPL Aware Node (RAN) to
> advertise a
>   438       selection of its candidate neighbors as siblings to the
> Root, more in
>   439       Section 6.4.  The sibling selection process is out of scope.
> The
>   440       expectation is that a node reports a Sibling Address in a
> SIO based
>   441       on an active address registration [RFC8505] from that
> sibling for
>   442       that address with the 'R' flag not set in the EARO.  The
> node may
>   443       assess the liveliness of the sibling at any time by
> performing a
>   444       registration for one of its own addresses, either a link
> local or a
>   453       global one, depending on whether the node expects the
> sibling to
>   454       perform a matching advertisement in its own SIO.
> 
> I am not even going to try to understand this without a prior graphical
> example use case on how this would be used.
> 
>   456    3.3.  P-DAO Request
>   457
>   458       Two new RPL Control Messages are also introduced, to enable
> a RAN to
>   459       request the establishment of a Track between self as the
> Track
>   460       Ingress Node and a Track Egress.  The RAN makes its request
> by
>   461       sending a new P-DAO Request (PDR) Message to the Root.  The
> Root
>   462       confirms with a new PDR-ACK message back to the requester
> RAN, see
>   463       Section 6.1 for more.
> 
> The split between this text and 6.1 is weird and IMHO not helpfull. Merge.
> WHats ven the logic to split some explanations out into section 6 and leave
> others (P-DAO for example) here ?

The logic is imposed on us by the "update rfc-blah" ways. The update is gone but we still need to summarize what we extend.
I'm trying to minimize what goes in the extend "sections", you'll see a lot of reshuffling

> 
> I would also like some motivational example of what would make a RAN send
> out a P-DAO Request. Could it be that the RAN is a PSE ingres ? I am
> asking,
> because this looks like a request where it might be highly desirable to
> have
> more contextual information in the request than just the few fields defined
> in
> 6.1. How about a 32 bit policy-ID field or the like.. But just guessing.
> Something for the PCE to match up PSE stuff it established with P-DAO
> requests it receives later. Unless you feel strongly that association is
> unambiguous now.

I agree with the intention, though this can get quite complex quite quickly so we kept it explicitly out of scope.
I suspect we'll need a new TLV for that sort of control. There could be multiple flows, as many descriptors

> 
>   463     A positive PDR-ACK indicates that the Track
>   464       was built and that the Roots commits to maintain the Track
> for the
>   465       negotiated lifetime.  In the case of a complex Track, each
> Segment is
>   466       maintained independently and asynchronously by the Root,
> with its own
>   467       lifetime that may be shorter, the same, or longer than that
> of the
>   468       Track.
> 
> I guess those different liftimes are not meant to make the solution more
> fragile
> by randomnly expiring timers for different parts of a track, but because
> the
> the parts of a projected routes affected can only change when they expire ?
> In any case, an explanation of the semantic impact of lifetimes would be
> useful.

Once you tear down a segment to replace it, everything is desynchronized. 

> 
>   468     The Root may use an asynchronous PDR-ACK with an negative
>   469       status to indicate that the Track was terminated before its
> time.
> 
> And if that happens, what should a RAN do that has some configuration
> telling
> it to request the P-DAO ? Sounds like the need for the definition of some
> exponential back-off re-request scheme ?

I added a neg  status values to indicate transient - retry later
Other values indicates permanent. Pe your other request I move the text to the P DAO RAQ section.
It now looks like

5.1.  New P-DAO Request Control Message

  The P-DAO Request (PDR) message is sent by a Node in the main DODAG
  to the Root.  It is a request to establish or refresh a Track where
  this node is Track Ingress, and signals whether an acknowledgment
  called PDR-ACK is requested or not.  A positive PDR-ACK indicates
  that the Track was built and that the Roots commits to maintain the
  Track for the negotiated lifetime.

  The Root may use an asynchronous PDR-ACK with an negative status to
  indicate that the Track was terminated before its time.  A status of
  "Transient Failure" (see Section 10.9) is an indication that the PDR
  may be retried after a reasonable time that depends on the
  deployment.  Other negative status values indicate a permanent error;
  the tentative must be abandoned until a corrective action is taken at
  the application layer or through network management.

> Also, when the request is just renewed upon timeout, but the reply changes,
> i figure that the impact in stateful mode may be some inconsistent
> forwarding
> while the track midpoints update their routes. No issue i guess in
> stateless.
> Would be useful to mention. Not sure if/what countermeasures for
> consistency
> RPL already provides. If it does, would be good to mention.

This would be guessing. I'm not sure I'd know what to write.

> 
>   471    3.4.  Extending the RPI
>   472
>   473       Sending a Packet within a RPL Local Instance requires the
> presence of
>   474       the abstract RPL Packet Information (RPI) described in
> section 11.2.
>   475       of [RPL] in the outer IPv6 Header chain (see
> [USEofRPLinfo]).  The
>   476       RPI carries a local RPLInstanceID which, in association with
> either
>   477       the source or the destination address in the IPv6 Header,
> indicates
>   478       the RPL Instance that the packet follows.
>   479
>   480       This specification extends [RPL] to create a new flag that
> signals
>   481       that a packet is forwarded along a projected route.
>   482
>   483       Projected-Route 'P':  1-bit flag.  It is set to 1 if this
> packet is
>   484          sent over a projected route and set to 0 otherwise.
> 
> This leaves me guessing when and how this applied. I can see how this would
> be
> applicable to both stateful and stateless Projected route operastions, but
> in
> either case, the forwarding plane behavior needs to be explained, ideall
> with
> example for each (of the two?) cases.

I clarified that it is set in the RPI in the IPinIP encapsulation and not mutable.

  Projected-Route 'P':  1-bit flag.  It is set to 1 in the RPI that is
     added in the encapsulation when a packet is sent over a Track.  It
     is set to 0 when a packet is forwarded along the main Track,
     including when the packet follows a Segment that joins loose hops
     of the main DODAG.  The flag is not mutable en-route.

  The encoding of the 'P' flag in native format is shown in Section 4.2
  while the compressed format is indicated in Section 4.3.
> 
> Also: Please add forward pointer to next section saying "For concrete
> encoding
> of the P flag, see section 4.".

Sure, see above

> 
>   486    4.  Extending RFC 6553
> 
> I think this RFC MUST claim to be an update to RFC6553, because the P-flag
> added to this concrete RPI RFC exhausts a non-IANA extension point, and
> the only way to formally avoid that any other RFCs could collide (and
> allocate
> the same bit) is to make this RFC an update to RFC6553. Same logic also
> applies to update for RFC6550 and P-DAO P flag.

Raising this to Alvaro's attention. This update thingy is always a tight discussion with him.

> 
> Just to be save, claim also update to RFC9008 to given how you mention it,
> and we obviously want this RFC to also apply to RPI headers with 0x23.
> 
>   511                                         +-+-+-+-+-+-+-+-+-+-+-+-+-
> +-+-+-+
>   512                                         |  Option Type  |  Opt
> Data Len |
>   513         +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-
> +-+-+-+
>   514         |O|R|F|P|0|0|0|0| RPLInstanceID |          SenderRank
> |
>   515         +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-
> +-+-+-+
>   516         |                         (sub-TLVs)
> |
>   517         +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-
> +-+-+-+
>   518
>   519                        Figure 2: Extended RPL Option Format
>   520
>   521       Option Type:  0x23 or 0x63, see [USEofRPLinfo]
>   522
>   523       Opt Data Len:  See [RFC6553]
>                         ^ unchanged,
>   524
>   525       'O', 'R' and 'F' flags:  See [RFC6553].  Those flags MUST be
> set to 0
>   526          by the sender and ignored by the receiver if the 'P' flag
> is set.
>   527
>   528       Projected-Route 'P':  1-bit flag as defined in Section 3.4.
>   529
>   530       RPLInstanceID:  See [RFC6553].  Indicates the TrackId if the
> 'P' flag
>   531          is set.
> 
> Add 'see also section 3.1' ?

Sure

> 
>   533       SenderRank:  See [RFC6553].  This field MUST be set to 0 by
> the
>   534          sender and ignored by the receiver if the 'P'flag is set.
> 
> Would you mind to add expanation as to what can happen when this passes via
> a router not supporting
> this RFC and therefore ignoring the P flag but interpreting the other
> fields ?

This cannot happen. RPL routers that forward along an instance must understand that instance. 
That was already true with RFC 6550. A Track is actually a RPL instance.

> 
>   536    5.  Extending RFC 8138
>   537
>   538       Section 6.3 of [RFC8138] presents the formats of the 6LoWPAN
> Routing
>   539       Header of type 5 (RPI-6LoRH) that compresses the RPI for
> normal RPL
>                                       ^
>   in the IANA "Critical 6LoWPAN Routing Header Type Registry"
> 
>   540       operation.  The format of the RPI-6LoRH is not suited for
> Projected
>   541       routes since the O,R,F flags are not used and the Rank is
> unknown and
>   542       ignored.
>   543
>   544       This specification introduces a new 6LoRH, the P-RPI-6LoRH,
> with a
>   545       type of 7.  The P-RPI-6LoRH header is usually a a Critical
> 6LoWPAN
>                    ^                                   ^^^
>                    |
>   in the IANA "Critical 6LoWPAN Routing Header Type Registry"
> 
>   546       Routing Header, but it can be elective as well if an SRH-
> 6LoRH is
>                                      ^^^^^^^^^^^^^^^^^^^
> also use type TBD1 from the IANA "Electivce 6LoWPAN Routing Header Type
> Registry"
> 
> https://www.iana.org/assignments/_6lowpan-parameters/_6lowpan-
> parameters.xhtml#critical-6lowpan-routing-header-type
> 
> Elective Type 7 is already gone. If you want to get the same numbers for
> elective and critical yo may want to also revisit to choose 7 for critical.
> 
> Who made you be so adventerous to write an actual number 7 into the draft
> without an
> early allocation anyhow ? How about asking for an early allocation ?

Thanks for the catch : ) No need for early allocation. I added (suggested) and moved to 8

> 
>   547       present and controls the routing decision.
>   548
>   549       The P-RPI-6LoRH is designed to compress the RPI along RPL
> Projected
>   550       Routes.  It sformat is as follows:
>                     ^^^
>   551
>   565                    0                   1                   2
>   566                    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3
>   567                   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-
> +
>   568                   |1|0|E| Length  | 6LoRH Type 7  | RPLInstanceID
> |
>   569                   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-
> +
>   570
>   571
>   572                            Figure 3: P-RPI-6LoRH Format
>   573
>   574       Elective 'E':  See [RFC8138].  The 'E' flag is set to 1 to
> indicate
>   575          an Elective 6LoRH, meaning that it can be ignored when
> forwarding.
> 
> Please explain length or point to RFC8138 (inchanged from RFC8318, section
> XX.YY).

Added  in 4.3.  Extending RFC 8138

  The 6LoWPAN Routing Header [RFC8138] specification introduces a new
  IPv6 over Low-Power Wireless Personal Area Network (6LoWPAN)
  [RFC6282] dispatch type for use in 6LoWPAN route-over topologies,
  which initially covers the needs of RPL data packet compression.

  Section 4 of [RFC8138] presents the generic formats of the 6LoWPAN
  Routing Header (6LoRH) with two forms, one Elective that can be
  ignored and skipped when the router does not understand it, and one
  Critical which causes the packet to be dropped when the router cannot
  process it.  The 'E' Flag in the 6LoRH indicates its form.  In order
  to skip the Elective 6LoRHs, their format imposes a fixed expression
  of the size, whereas the size of a Critical 6LoRH may be signaled in
  variable forms to enable additional optimizations.
> 
> Please DO NEVER include the assicned number into the field, just write
> "6LoRH Type".
> Explain in text that is the assined type for criical or elective depending
> on E flag.
> 

Now reads as 

  This specification introduces a new 6LoRH, the P-RPI-6LoRH that can
  be used in either Elective or Critical 6LoRH form, see Table 21 and
  Table 22 respectively.  The new 6LoRH MUST be used as a Critical
  6LoRH, unless an SRH-6LoRH is present and controls the routing
  decision, in which case it MAY be used in Elective form.

  The P-RPI-6LoRH is designed to compress the RPI along RPL P-Routes.
  Its format is as follows:

               0                   1                   2
               0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3
              +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
              |1|0|E| Length  |  6LoRH Type   | RPLInstanceID |
              +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

                       Figure 5: P-RPI-6LoRH Format

  Type:  IANA is requested to define the same value of the type for
     both Elective and Critical forms.  A type of 8 is suggested.

  Elective 'E':  See [RFC8138].  The 'E' flag is set to 1 to indicate
     an Elective 6LoRH, meaning that it can be ignored when forwarding.

> Explain that RPLInstanceID is the TrackID (see section 3.1 ?) Or if not,
> what it is.

Done, pointing to the relevant section after all the reshuffling

Looks like 

  TrackID:  8-bit field.  In the context of this specification, the
     TrackID field signals the RPLInstanceID of the DODAG formed by the
     Track, see Section 3.3 and Section 6.2.

> 
>   577    6.  New RPL Control Messages and Options
>   578
>   579    6.1.  New P-DAO Request Control Message
>   580
>   581       The P-DAO Request (PDR) message is sent by a Node in the
> main DODAG
>   582       to the Root.  It is a request to establish or refresh a
> Track where
>   583       this node is Track Ingress.  The source IPv6 address of the
> PDR
>   584       signals the Track Ingress of the requested Track, and the
> TrackID is
>   585       indicated in the message itself.  One and only one RPL
> Target Option
>   586       MUST be present in the message.  The RTO signals the Track
> Egress,
>   587       more in Section 7.1.
>   588
>   589       The RPL Control Code for the PDR is 0x09, to be confirmed by
> IANA.
> 
> Correct form is TBDi (TBD2 ?) instead of 0x09.
> If you are not in a hurry, all TBDi can be resolved during IANA/RFC-editor
> procss.
> If you want to lock in a number get an early allocation first.

We want interoperable early implementations. It does not hurt to suggest does it?

> 
>   590       The format of PDR Base Object is as follows:
>   591
>   592          0                   1                   2
> 3
>   593          0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8
> 9 0 1
>   594         +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-
> +-+-+-+
>   595         |   TrackID     |K|R|   Flags   |  ReqLifetime  |
> PDRSequence   |
>   596         +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-
> +-+-+-+
>   597         |   Option(s)...
>   598         +-+-+-+-+-+-+-+-+
>   599
>   600                         Figure 4: New P-DAO Request Format
>   601
>   602       TrackID:  8-bit field indicating the RPLInstanceID
> associated with
>   603          the Track.
>   604
>   605       K:  The 'K' flag is set to indicate that the recipient is
> expected to
>   606          send a PDR-ACK back.
>   607
>   608       R:  The 'R' flag is set to request a Complex Track for
> redundancy.
>   609
>   610       Flags:  Reserved.  The Flags field MUST initialized to zero
> by the
>   611          sender and MUST be ignored by the receiver
>   612
>   613
>   614
>   615
>   616    Thubert, et al.          Expires 28 January 2022
> [Page 11]
>   617
> 
> 
>   618    Internet-Draft               DAO Projection
> July 2021
>   619
>   620
>   621       ReqLifetime:  8-bit unsigned integer.  The requested
> lifetime for the
>   622          Track expressed in Lifetime Units (obtained from the
> DODAG
>   623          Configuration option).
>   624
>   625          A PDR with a fresher PDRSequence refreshes the lifetime,
> and a
>   626          PDRLifetime of 0 indicates that the track should be
> destroyed.
> 
> ... It would be sent for example when the function for which the Node
> requested
> the track is deactivated. ?!

True. Added "                                 e.g., when the
             application that requested the Track terminates. "

> 
>   628       PDRSequence:  8-bit wrapping sequence number, obeying the
> operation
>   629          in section 7.2 of [RPL].  The PDRSequence is used to
> correlate a
>   630          PDR-ACK message with the PDR message that triggered it.
> It is
>   631          incremented at each PDR message and echoed in the PDR-ACK
> by the
>   632          Root.
>   633
>   634    6.2.  New PDR-ACK Control Message
>   635
>   636       The new PDR-ACK is sent as a response to a PDR message with
> the 'K'
>   637       flag set.  The RPL Control Code for the PDR-ACK is 0x0A, to
> be
>   638       confirmed by IANA.  Its format is as follows:
> 
> Same IANA/TBD concern as above.

Well yes; same requirement too ; )

> 
>   639
>   640          0                   1                   2
> 3
>   641          0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8
> 9 0 1
>   642         +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-
> +-+-+-+
>   643         |    TrackID    |     Flags     | Track Lifetime|
> PDRSequence  |
>   644         +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-
> +-+-+-+
>   645         | PDR-ACK Status|                Reserved
> |
>   646         +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-
> +-+-+-+
>   647         |  Option(s)...
>   648         +-+-+-+-+-+-+-+
>   649
>   650                    Figure 5: New PDR-ACK Control Message Format
>   651
>   652       TrackID:  The RPLInstanceID of the Track that was created.
> The value
>   653          of 0x00 is used to when no Track was created.
>                              ^^
> 
> Same question as in before. Any standardized logic what to do in case of
> 0x00 failure,
> if not, then why not.

This was a leftover when any node could create a Track starting at this node.
Now only the ingress can issue a PDR, and it must provide a trackID.

> 
>   654
>   655       Flags:  Reserved.  The Flags field MUST initialized to zero
> by the
>   656          sender and MUST be ignored by the receiver
>   657
>   658       Track Lifetime:  Indicates that remaining Lifetime for the
> Track,
>   659          expressed in Lifetime Units; the value of zero (0x00)
> indicates
>   660          that the Track was destroyed or not created.
>   661
>   662       PDRSequence:  8-bit wrapping sequence number.  It is
> incremented at
>   663          each PDR message and echoed in the PDR-ACK.
> 
> according to section 7.2 of [RPL] ?!
> 
>   665       PDR-ACK Status:  8-bit field indicating the completion.  The
> PDR-ACK
>                                                                 ^ status
>   666          Status is substructured as indicated in Figure 6:
>   676
>   677                                     0 1 2 3 4 5 6 7
>   678                                    +-+-+-+-+-+-+-+-+
>   679                                    |E|R|  Value    |
>   680                                    +-+-+-+-+-+-+-+-+
>   681
>   682                           Figure 6: PDR-ACK status Format
>   683
>   684          E:  1-bit flag.  Set to indicate a rejection.  When not
> set, the
>   685             value of 0 indicates Success/Unqualified acceptance
> and other
>   686             values indicate "not an outright rejection".
>   687          R:  1-bit flag.  Reserved, MUST be set to 0 by the sender
> and
>   688             ignored by the receiver.
>   689          Status Value:  6-bit unsigned integer.  Values depending
> on the
>   690             setting of the 'E' flag, see Table 27 and Table 28.
>   691
>   692       Reserved:  The Reserved field MUST initialized to zero by
> the sender
>                                             ^ be
>   693          and MUST be ignored by the receiver
>   694
>   695    6.3.  Via Information Options
>   696
>   697       A VIO signals the ordered list of IPv6 Via Addresses that
> constitutes
>   698       the hops of either a Serial Track or a Segment of a more
> Complex
>   699       Track.  A VIO MUST contain at least one Via Address, and a
> Via
>   700       Address MUST NOT be present more than once, otherwise the
> VIO MUST be
>   701       ignored.
> 
> If i understand it correctly, for the SR-VIO, the sequence of addresses
> would
> ultimately end up in the RPL steering header of packets. I browsed through
> RFC8754 and think i can not find a similar exclusion. So a justification
> why this optimization is done for SF-VIO would be good if you really want
> to keep it.

If you find the same address twice you made a loop. There is already logic to avoid that in several protocols.
Now the rule to ignore the VIO requires additional validation code that the LLN node is node necessarily ready to pay for.
Suggested text:

                                                      The VIO contains
     one or more addresses listed in the order of progression of the
     packet from Ingress to Egress.  In a No-Path Non-Storing Mode
     P-DAO, the list MUST be elided from the NSM-VIO since the state in
     the Ingress is erased regardless.  In all other cases, a VIO MUST
     contain at least one Via Address, and a Via Address MUST NOT be
     present more than once, which would create a loop.

     A node that processes a VIO MAY verify whether one of these
     conditions happen, and when so, it MUST ignore the P-DAO and
     reject it with a RPL Rejection Status of "Error in VIO" in the
     DAO-ACK, see Section 10.14.

> 
> For SF-VIO, this seems to be taken apart and every node listed creates a
> next-hop route. So if the same node is addressed twice, it could become
> confused, which instance of its own address to choose to install the route,
> right ?
> 

Right

> 
> And what does "ignore" mean anyhow, e.g.: would the Target Options still be
> acted
> upon ? It rather sounds to me as if you would want the whole Projected DAO
> message
> to be rejected ?

There's nothing else to act upon. The Targets are the destinations of the routes via the VIO.
Still I incorporated the clarification in the text above.

> 
> Is duplicate address the only case of a "broken" projected DAO that you can
> discover and reject ? E.g.: can nodes have multiple addresses ? Such as
> anycast addresses ?
> 

Added that the addresses must be ULA or GUA. Other checks are listed in the new text 

> Would be good to elaborate about such "broken" projected DAO more if there
> are different
> cases of it.

Did some but note that an LLN node will probably NOT od the checks, due to constrained code space.
The MUST really applies to the Root that forms the VIO.

> 
>   701       The format of the Via Information Options is as follows:
>   732
>   733            0                   1                   2
> 3
>   734            0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7
> 8 9 0 1
>   735           +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-
> +-+-+-+-+
>   736           |   Type        | Option Length |     Flags     |
> SegmentID   |
>   737           +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-
> +-+-+-+-+
>   738           |Segm. Sequence | Seg. Lifetime |      SRH-6LoRH header
> |
>   739           +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-
> +-+-+-+-+
>   740           |
> |
>   741           +
> +
>   742           .
> .
>   743           .                     Via Address 1
> .
>   744           .
> .
>   745           +
> +
>   746           |
> |
>   747           +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-
> +-+-+-+-+
>   748           |
> |
>   749           .                              ....
> .
>   752           |
> |
>   751           +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-
> +-+-+-+-+
>   752           |
> |
>   753           +
> +
>   754           .
> .
>   755           .                     Via Address n
> .
> 
> add (n <= 15) ?

Why 15? Though this seems very large, SR-6LoRH may compress that to 30 bytes.
RFC 6554 does not provide an explicit limit.

> 
>   756           .
> .
>   757           +
> +
>   758           |
> |
>   759           +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-
> +-+-+-+-+
>   760
>   761
>   762                      Figure 7: VIO format (uncompressed form)
>   763
>   764       Option Type:  0x0B for SF-VIO, 0x0C for SR-VIO (to be
> confirmed by
>   765          IANA)
> 
> Same concern as prior illegal occupations of IANA codepoints in this spec
> ;-))

Same answer I guess; note that RPL AODV will ship prior to this, so I'm pushing the suggested values.

> 
>   767       Option Length:  In bytes; variable, depending on the number
> of Via
>   768          Addresses and the compression.
> 
> Something like total size of the Option ... in bytes ?

Done 

> Is this semantic mandated by prior RFC ?  

RFC 6550, note that the first 2 bytes are not counted

> If it could count differently,
> it could allow for more than 15 hops, although i have no idea if that would
> be relevant.

Why 15? 

> 
>   770       SegmentID:  8-bit field that identifies a Segment within a
> Track or
>   771          the main DODAG as indicated by the TrackID field.  The
> value of 0
>   772          is used to signal a Serial Track, i.e., made of a single
> segment.
> 
> Another basic signaling element not yet explained, so somewhat mysterious
> here

Reordered as you suggested
> 
>   774       Segment Sequence:  8-bit unsigned integer.  The Segment
> Sequence
>   775          obeys the operation in section 7.2 of [RPL] and the
> lollipop
>   776          starts at 255.
>   777
>   778          When the Root of the DODAG needs to refresh or update a
> Segment in
>   779          a Track, it increments the Segment Sequence individually
> for that
>   780          Segment.
>   781
>   789          The Segment information indicated in the VIO deprecates
> any state
>   790          for the Segment indicated by the SegmentID within the
> indicated
>   791          Track and sets up the new information.
>   792
>   793          A VIO with a Segment Sequence that is not as fresh as the
> current
>   794          one is ignored.
>   795
>   796          A VIO for a given DODAGID with the same (TrackID,
> SegmentID,
>   797          Segment Sequence) indicates a retry; it MUST NOT change
> the
>   798          Segment and MUST be propagated or answered as the first
> copy.
>   799
>   800       Segment Lifetime:  8-bit unsigned integer.  The length of
> time in
>   801          Lifetime Units (obtained from the Configuration option)
> that the
>   802          Segment is usable.
>   803
>   804          The period starts when a new Segment Sequence is seen.
> The value
>   805          of 255 (0xFF) represents infinity.  The value of zero
> (0x00)
>   806          indicates a loss of reachability.
>   807
>   808          A P-DAO message that contains a VIO with a Segment
> Lifetime of
>   809          zero is referred as a No-Path P-DAO in this document.
>   810
>   811       SRH-6LoRH header:  The first 2 bytes of the (first) SRH-
> 6LoRH as
>   812          shown in Figure 6 of [RFC8138].  A 6LoRH Type of 4 means
> that the
>   813          VIA Addresses are provided in full with no compression.
>   814
>   815       Via Address:  An IPv6 address along the Segment.
>   816
>   817          In a SF-VIO, the list is a strict path between direct
> neighbors,
>   818          from the Segment Ingress to Egress, both included.  The
> router
>   819          that processes an SF-VIO MAY create additional routing
> state
>   820          towards the nodes after self via the node immediately
> after self
>   821          in the SF-VIO, but in case of memory shortage the routes
> to the
>   822          Targets have precedence since they are the ones that the
> router
>   823          commits to store.
> 
> Again, i am at a lack of understanding the base theory of operations of
> forwarding,
> I have Track Ingres, Track Egres, Destinations, Target and somehow i have
> Segments with addresses, wheree i do not know how Segment Ingres and
> Segment Egres
> may or may not map to Track Ingres/Egres, Destinations or Targets.
> 
> Picture me this, please ? ;-))

Done! It now looks like this and is early in the doc, moved to new section 3 after the terminology as opposed to within the terminology.

          CPF               CPF          CPF                 CPF

                         Southbound API

     _-._-._-._-._-._-._-._-._-._-._-._-._-._-._-._-._-._-._-._-._-._-
   _-._-._-._-._-._-._-._-._-._-._-._-._-._-._-._-._-._-._-._-._-._-

                        +----------+
                        | RPL Root |
                        +----------+
                          (      )
                (                                  )
          (              DODAG                              )
            (                                           )
    (                                                         )
                                                                    )
    <-    Leg 1            B,                            E ->
    <--- Segment 1 A,B ---> <------- Segment 2 C,D,E ------->

               FWD  --z  Relay --z   FWD  --z   FWD          Target 1
           z-- Node  z--  Node  z--  Node  z--  Node --z     /
        --z    (A)        (B) \      (C)        (D)  z--    /
  Track                        \                       Track
  Ingress                    Segment 5                 Egress - Tgt 2
    (I)                           \                     (E)
        --z                        \                 z--    \
         z-- FWD   --z  FWD  --z  Relay --z  FWD  --z        \
             Node   z-- Node  z-- Node   z-- Node            Target 3
             (F)        (G)       (H)        (J)

    <------ Segment 3 F,G,H ------> <---- Segment 4 J,E ---->
    <-      Leg 2                  H,                    E ->

    <--- Segment 1 A,B ---> <- S5-> <---- Segment 4 J,E ---->
    <-      Leg 3          B,      H,                    E ->
                                                                    )
     (
                (                                        )

                      Figure 2: Segments and Tracks

> 
> So, what exactly does the routing state look like ? whats the lookup ?
> Something like (RPLInstanceID==TrackID, Destination) ? Or rather: What is
> the example scenario when a the above described additional routing state
> to the nodes after self would would be used ?

All in section 3 now

> 
>   825          In an SR-VIO, the list includes the egress but not the
> ingress
>   826          node.  It starts at the first hop and ends at a Track
> Egress.  In
> 
> Seems like a mayor difference in flexiblity of stateful vs. stateless
> operations than ?
> No segment layer with SR-VIO ?

The segments are strict and join the loose SRH hops. See the SRH hops as DODAG edges (signaled as NSM) and Segments as vertices (signaled as SM).

> 
>   827          that case, the Track Egress MUST be considered as an
> implicit
>   828          Target, so it MUST NOT be listed it in a RPL Target
> Option.  The
>   829          list in an SR-VIO may be loose, provided that each listed
> node has
>   830          a path to the next listed node, e.g., via a segment or
> another
>   831          Track.
> 
> But you are not telling me you would want to create the freedom to have
> stateless
> operation rely on routing state created by SF-VIO where you said it might
> not exist in case of memory shortage which the stateless encapsulator would
> know nothing about, right ?

The root should install the segments first and then join them with a track.

> 
>   833          In the case of a SF-VIO, or if [RFC8138] is not used in
> the data
>   834          packets, then the Root MUST use only one SRH-6LoRH per
> Via
>   835          Information Option, and the compression is the same for
> all the
>   836          addresses, as shown in Figure 7.
> 
> I do not see any address compression in Figure 7, nor was compression
> mentioned
> so far.

It's there but non obvious. The SRH-6LoRH header field is the beginning of it.

> 
> Do you mean figure 7 in section 5.1 of [RFC8138] ?

I'll make it more visible.

    0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |   Type        | Option Length |     Flags     |   SegmentID   |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |Segm. Sequence | Seg. Lifetime |      SRH-6LoRH header         |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                                                               |
   .           Via Address 1 (compressed by RFC 8138)              .
   |                                                               |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                                                               |
   .                              ....                             .
   |                                                               |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                                                               |
   .           Via Address n (compressed by RFC 8138)              .
   |                                                               |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

> 
>   845          In case of an SR-VIO, and if [RFC8138] is in use in the
> main
>   846          DODAG, then the Root SHOULD optimize the size of the SR-
> VIO;
> 
> Any example how ?
> 
>   846         more
>   847          than one SRH-6LoRH may be present, e.g., if the
> compression level
>   848          changes inside the Segment and different SRH-6LoRH Types
> are
>   849          required.
> 
> That rather sounds as if those are cases where optimiation would no be
> possible ?

Changed to 

                                 the Root SHOULD optimize the size of the
             NSM-VIO; this means that more than one SRH-6LoRH may be present,
             if the compression level changes inside the Segment and using
             different SRH-6LoRH Types make the VIO shorter.

> 
>   849        The content of the SR-VIO starting at the first SRH-
>   850          6LoRH header is thus verbatim the one that the Track
> Ingress
>   851          places in the packet encapsulation to reach the Track
> Ingress.
> 
> Example with picture would help.

Added

6.7.  Encapsulation and Compression

  When using [RFC8138] in the main DODAG operated in Non-Storing Mode
  in a 6LoWPAN LLN, a typical packet that circulates in the main DODAG
  is formatted as shown in Figure 13, representing the case where :

  +-+ ... -+- ... -+- ... -+-+- ... +-+-+-+ ... +-+-+ ... -+ ... +-...
  |11110001|  SRH- | RPI-  | IP-in-IP | NH=1      |11110CPP| UDP | UDP
  | Page 1 | 6LoRH | 6LoRH |  6LoRH   |LOWPAN_IPHC| UDP    | hdr |Payld
  +-+ ... -+- ... -+- ... -+-+- ... +-+-+-+ ... +-+-+ ... -+ ... +-...
                                       <=        RFC 6282      =>
            <================ Inner packet ==================== = =

          Figure 13: A Packet as Forwarded along the Main DODAG

  Since there is no page switch between the encapsulated packet and the
  encapsulation, the first octet of the compressed packet that acts as
  page selector is actually removed at encapsulation, so the inner
  packet used in the descriptions below start with the SRH-6LoRH, and
  is verbatim the packet represented in Figure 13 from the second octet
  on.

  When encapsulating that inner packet to place it in the Track, the
  first header that the Ingress appends at the head of the inner packet
  is an IP-in-IP 6LoRH Header; in that header, the encapsulator
  address, which maps to the IPv6 source address in the uncompressed
  form, contains a GUA or ULA IPv6 address of the Ingress node that
  serves as DODAG ID for the Track, expressed in the compressed form
  and using the DODAGID of the main DODAG as compression reference.  If
  the address is compressed to 2 bytes, the resulting value for the
  Length field shown in Figure 14 is 3, meaning that the SRH-6LoRH as a
  whole is 5-octets long.

     0                   1                   2
     0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-     ...     -+
    |1|0|1| Length  | 6LoRH Type 6  |  Hop Limit    | Track DODAGID |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-     ...     -+

                   Figure 14: The IP-in-IP 6LoRH Header

  At the head of the resulting sequence of bytes, the track Ingress
  then adds the RPI that carries the TrackID as RPLinstanceID as a P-
  RPI-6LoRH Header, as illustrated in Figure 5, using the TrackID as
  RPLInstanceID.  Combined with the IP-in-IP 6LoRH Header, this allows
  to identify the Track without ambiguity.

  The SRH-6LoRH is then added at the head of the resulting sequence of
  bytes as a verbatim copy of the content of the SR-VIO that signaled
  the selected Track Leg.

       0                   1
       0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5  ..         ..        ..
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-    -+-    -+ ... +-    -+
      |1|0|0|  Size   |6LoRH Type 0..4| Hop1 | Hop2 |     | HopN |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-    -+-    -+ ... +-    -+
                                                Where N = Size + 1

                     Figure 15: The SRH 6LoRH Header

  The resulting encapsulated packet looks as illustrated in Figure 16:

  +-+ ... -+-+-+- ... -+-+-+- ... -+-+-+-+-+- ... +-+-+-+-+-+-+- ...
  | Page 1 |  SRH-6LoRH  | P-RPI-6LoRH | IP-in-IP 6LoRH | Inner Packet
  +-+ ... -+-+-+- ... -+-+-+- ... -+-+-+-+-+- ... +-+-+-+-+-+-+- ...

   Signals :  Loose Hops :    TrackID  :  Track DODAGID :

          Figure 16: A Packet as Forwarded along a Track

> 
>   853    6.4.  Sibling Information Option
>   854
>   855       The Sibling Information Option (SIO) provides indication on
> siblings
>   856       that could be used by the Root to form Projected Routes.
> One or more
>   857       SIO(s) may be placed in the DAO messages that are sent to
> the Root in
>   858       Non-Storing Mode.
> 
> Is this some poor-mans (little signaling) form of signaling to help the
> root do some topology discovery ? If so, then it probably would be
> something
> that needs to go to the PCE, right ? Maybe this type of high level
> explanation in the tex would be helpful. If i am guessing wrong then i even
> understand
> less.

You're correct

Added

   This specification expects that the main RPL Instance is operated in RPL
   Non-Storing Mode to sustain the exchanges with the Root. Based on its
   comprehensive knowledge of the parent-child relationship, the Root can form
   an abstracted view of the whole DODAG topology. This document adds the
   capability for nodes to advertise additional sibling information to
   complement the topological awareness of the Root to be passed on to the PCE.

> 
>   859
>   860       The format of the SIO is as follows:
>   861
>   862            0                   1                   2
> 3
>   863            0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7
> 8 9 0 1
>   864           +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-
> +-+-+-+-+
>   865           |   Type        | Option Length |Comp.|B|D|Flags|
> Opaque     |
>   866           +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-
> +-+-+-+-+
>   867           |            Step of Rank       |          Reserved
> |
>   868           +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-
> +-+-+-+-+
>   869           |
> |
>   870           +
> +
>   871           .
> .
>   872           .       Sibling DODAGID (if the D flag not set)
> .
>   873           .
> .
>   874           +
> +
>   875           |
> |
>   876           +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-
> +-+-+-+-+
>   877           |
> |
>   878           +
> +
>   879           .
> .
>   880           .                     Sibling Address
> .
>   881           .
> .
>   882           +
> +
>   883           |
> |
>   884           +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-
> +-+-+-+-+
>   885
>   886                    Figure 8: Sibling Information Option Format
>   887
>   888       Option Type:  0x0D (to be confirmed by IANA)
> 
> You're maxing out your repeat offender penalty of illegal IANA code point
> misappropriation ;-)

Not illegal, just suggested, added (to be confirmed by IANA)

> 
>   890       Option Length:  In bytes, the size of the option.
>   891
>   892       Compression Type:  3-bit unsigned integer.  This is the SRH-
> 6LoRH
>   901          Type as defined in figure 7 in section 5.1 of [RFC8138]
> that
>   902          corresponds to the compression used for the Sibling
> Address and
>   903          its DODAGID if  resent.  The Compression reference is the
> Root of
>                             ^ p ?
>   904          the main DODAG.
>   905
>   906       Reserved for Flags:  MUST be set to zero by the sender and
> MUST be
>   907          ignored by the receiver.
>   908
>   909       B:  1-bit flag that is set to indicate that the connectivity
> to the
>   910          sibling is bidirectional and roughly symmetrical.  In
> that case,
>   911          only one of the siblings may report the SIO for the hop.
> If 'B'
> 
> How would they do that (decide which one reports the SIO ?).
> 
>   911          only one of the siblings may report the SIO for the hop.
> If 'B'
>   912          is not set then the SIO only indicates connectivity from
> the
>   913          sibling to this node, and does not provide information on
> the hop
>   914          from this node to the sibling.
> 
> I hope there is a simple way to do 'B' or more of a benefit than just
> cutting
> down reporting data by 50%. Any standardized mechanism ?*

OK, I remove that B flag for now, that's optimization. In the P-DAO discussion I added

                                                         If this router also
  registers an address to that sibling, and the link has similar properties in
  both directions, only the router with the lowest Interface ID in its
  registered address needs report the SIO, and the Root will assume symmetry.

> 
>   916       D:  1-bit flag that is set to indicate that sibling belongs
> to the
>   917          same DODAG.  When not set, the Sibling DODAGID is
> indicated.
>   918
>   919       Flags:  Reserved.  The Flags field MUST initialized to zero
> by the
>                                                  ^ be
>   920          sender and MUST be ignored by the receiver
>   921
>   922       Opaque:  MAY be used to carry information that the node and
> the Root
>   923          understand, e.g., a particular representation of the Link
>   924          properties such as a proprietary Link Quality Information
> for
>   925          packets received from the sibling.  An industrial
> Alliance that
>   926          uses RPL for a particular use / environment MAY redefine
> the use
>   927          of this field to fit its needs.
> 
> Is there a prior RPL example of such a field ? Pointer to that wold be
> useful. I hav a hard time seeing how this would not result in all type of
> misinterpretation unless the industrial alliance whose registration space
> is to be used for Upaque is unambigously derivable from the device (type)
> or link (type).
> 
> I would prefer to add MUST be set to 0 and be ignored upon receiving and
> go on saying, IETF specification MUST NOT use or define this field, but
> non-IETF specification may use it. Aka: why not give such external specs
> the same starting ground (0-filled) that we give our own extension points.
> 
> But again: pre-existing examples of BCP spec text for such an Opaque field
> would
> be helpful.

See RFC 8505 fig 1. 
The reason for specifying opaque is to ensure that it can be used by another party within their std that uses RPL, and there will not be an RFC later that reuses the field and creates a collision.

> 
>   929       Step of Rank:  16-bit unsigned integer.  This is the Step of
> Rank
>   930          [RPL] as computed by the Objective Function between this
> node and
>   931          the sibling.
>   932
>   933       Reserved:  The Reserved field MUST initialized to zero by
> the sender
>   934          and MUST be ignored by the receiver
>   935
>   936       Sibling DODAGID:  2 to 16 bytes, the DODAGID of the sibling
> in a
>   937          [RFC8138] compressed form as indicated by the Compression
> Type
>   938          field.  This field is present if and only if the D flag
> is not
>   939          set.
>   940
>   941       Sibling Address:  2 to 16 bytes, an IPv6 Address of the
> sibling, with
>   942          a scope that MUST be make it reachable from the Root,
> e.g., it
>   943          cannot be a Link Local Address.  The IPv6 address is
> encoded in
>   944          the [RFC8138] compressed form indicated by the
> Compression Type
>   945          field.
>   956
>   957       An SIO MAY be immediately followed by a DAG Metric
> Container.  In
>            ^ "A for consonant start of TLA"
>   958       that case the DAG Metric Container provides additional
> metrics for
>   959       the hop from the Sibling to this node.
>   960
>   961    7.  Projected DAO
> 
> Given how this section seems to describe what the title of the doc is,
> maybe it should also be named that way "Root initiated routing state".

I like that... Done

> 
>   962
>   963       This draft adds a capability to RPL whereby the Root of a
> main DODAG
> 
> Define what a "main DODAG" is. I guess its a DODAG not created by the
> mechanisms of this draft, but also: what are non-main DODAGs ? Or is the
> opposite just the projected DAO ?

Added in intro
"
  an external controller interacts with the RPL Root to compute centrally
  shorter Peer to Peer (P2P) paths within a pre-existing RPL Main DODAG.
"

Actually the system may recurse. A Track is a DODAG that may becomes a Main DODAG for tracks within tracks. But that takes us too far for this document.

> 
>   964       installs a Track as a collection of Projected Routes, using
> a
>   965       Projected-DAO (P-DAO) message to maintain each individual
> route.  The
> 
> What is an individual route ?

Changed to 

"
 for each individual Track Leg or Segment.
"
> 
>   966       P-DAO signals a collection of Targets in the RPL Target
> Option(s)
>   967       (RTO).  Those Targets can be reached via a sequence of
> routers
>   968       indicated in a VIO.  A P-DAO message MUST contain exactly
> one VIO,
>   969       which is either a SF-VIO or an SR-VIO, and MUST follow one
> or more
>   970       RTOs.  There can be at most one such sequence of RTO(s) and
> an Via
>   971       Information Option.  A track is identified by a tuple
> DODAGID,
>                             ^ in a P-DAO
> 
> Also, is "at most" correct ?
> How about there MUST be exactly one sequece of one or more RTO followed vy
> one VIO ?
> Or is it valid to have no RTO, or no VIO ?

I like your wording, simpler and to the point. 

> 
>   972       TrackID and each route within a Track is indexed by a
> SegmentID.
> 
> So, IMHO, this whole section 7 needs to come before section 3, because it
> is
> here where you start defining the functionality from higher layer, but then
> section 3 goes into the encoding details. I observe that all RFCs i
> remember
> have the order of starting wih the overall functionality and bring the
> encoding
> details of messages later (but then i am forgetfull of what i have read and
> also say this because as my comments before will show, i struggled  to make
> head & tails out of how the pieces fit together).
> 

I did reshuffle stuff quite a bit. Now it looks like
  1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   3
  2.  Terminology . . . . . . . . . . . . . . . . . . . . . . . . .   4
    2.1.  Requirements Language . . . . . . . . . . . . . . . . . .   4
    2.2.  References  . . . . . . . . . . . . . . . . . . . . . . .   4
    2.3.  Glossary  . . . . . . . . . . . . . . . . . . . . . . . .   4
    2.4.  Domain Terms  . . . . . . . . . . . . . . . . . . . . . .   5
  3.  Context and Goal  . . . . . . . . . . . . . . . . . . . . . .   6
    3.1.  RPL Applicability . . . . . . . . . . . . . . . . . . . .   6
    3.2.  RPL Routing Modes . . . . . . . . . . . . . . . . . . . .   7
    3.3.  On Tracks . . . . . . . . . . . . . . . . . . . . . . . .   8
    3.4.  Serial Track Signaling  . . . . . . . . . . . . . . . . .   9
      3.4.1.  Using Storing-Mode Segments . . . . . . . . . . . . .  10
      3.4.2.  Using Non-Storing-Mode joining Tracks . . . . . . . .  15
    3.5.  Complex Tracks  . . . . . . . . . . . . . . . . . . . . .  22
    3.6.  Scope and Expectations  . . . . . . . . . . . . . . . . .  24
  4.  Extending existing RFCs . . . . . . . . . . . . . . . . . . .  26
    4.1.  Extending RFC 6550  . . . . . . . . . . . . . . . . . . .  26
      4.1.1.  Projected DAO . . . . . . . . . . . . . . . . . . . .  26
      4.1.2.  Via Information Option  . . . . . . . . . . . . . . .  28
      4.1.3.  Sibling Information Option  . . . . . . . . . . . . .  28
      4.1.4.  P-DAO Request . . . . . . . . . . . . . . . . . . . .  28
      4.1.5.  Extending the RPI . . . . . . . . . . . . . . . . . .  29
    4.2.  Extending RFC 6553  . . . . . . . . . . . . . . . . . . .  29
    4.3.  Extending RFC 8138  . . . . . . . . . . . . . . . . . . .  30
  5.  New RPL Control Messages and Options  . . . . . . . . . . . .  31

>   973
>   974       A P-DAO MUST be sent from the address of the Root that
> serves as
>   975       DODAGID for the main DODAG.  It MUST be sent to a GUA or a
> ULA of
> 
> Expand GUA, ULA, provide reference for ULA, and/or add to initial list of
> terms in doc.

Added in first use and in acronyms

> 
>   976       either the ingress or the egress of the Segment, more below.
> If the
>   977       'K' Flag is present in the P-DAO, and unless the P-DAO does
> not reach
>   978       it, the ingress of the Segment is the node that acknowledges
> the
>   979       message, using a DAO-ACK that MUST be sent back to the
> address that
>   980       serves as DODAGID for the main DODAG.
> 
> Sentence too convoluted. Try to make two out of it.

What about

   If the 'K' Flag is present in the P-DAO, the P-DAO must be acknowledged
   using a DAO-ACK that is sent back to the address of the Root from which the
   P-DAO was received. In most cases, the first node of the Leg, Segment,
   or updated section of the Segment is the node that sends the acknowledgment.
   The exception to the rule is when an intermediate node in a Segment fails
   to forward a Storing Mode P-DAO to the previous node in the SM-VIO; in that
   case, the node that fails to forward the P-DAO sends the acknowledgment.
   The DAO-ACK MUST be sent back to the address that serves as DODAGID for the
   Main DODAG.
> 
> You are not using the same term here as in 974, so this makes one wonder if
> the DAO-ACK is to be sent back to the same address it was sent from or not.
> If it is meant to be sent back to the same address, why not say "sent back
> to
> the address of the root, which the P-DAO was sent from".

Adopted in the text above

> 
>   982       Like a classical DAO message, a P-DAO causes a change of
> state only
>   983       if it is "new" per section 9.2.2.  "Generation of DAO
> Messages" of
>   984       the RPL specification [RPL]; this is determined using the
> Segment
>   985       Sequence information from the VIO as opposed to the Path
> Sequence
>                                            ^ in the P-DAO
>   986       from a TIO.  Also, a Segment Lifetime of 0 in a VIO
> indicates that
>                     ^ in the DAO.
>   987       the projected route associated to the Segment is to be
> removed.
>   988
>   989       There are two kinds of operation for the Projected Routes,
> the
>   990       Storing Mode and the Non-Storing Mode.
>   991
>   992       *  The Non-Storing Mode is discussed in Section 7.3.2.  A
> Non-Storing
>   993          Mode P-DAO carries an SR-VIO with the loose list of Via
> Addresses
>                                                         ^ steering
>   994          that forms a source-routed Segment to the Track Egress.
> The
>   995          recipient of the P-DAO is the Track Ingress; it MUST
> install a
>   996          source-routed state to the Track Egress and reply to the
> Root
>   997          directly using a DAO-ACK message if requested to.
>   998
>   999       *  The Storing Mode is discussed in Section 7.3.1.  A
> Storing Mode
>  1000          P-DAO carries a  SF-VIO with the strict  list of Via
> Addresses from
>                             ^n                      ^ steering
>  1001          the ingress to the egress of the Segment in the data path
> order.
>  1002          The routers listed in the Via Addresses, except the
> egress, MUST
>  1003          install a routing state to the Target(s) via the next Via
> Address
>  1004          in the SF-VIO.  In normal operations, the P-DAO is
> propagated
>  1013          along the chain of Via Routers from the egress router of
> the path
>  1014          till the ingress one, which confirms the installation to
> the Root
>  1015          with a DAO-ACK message.
> 
> In 976 you point to further below as to where the P-DAO is to be sent. But
> hen
> in the bullet points 992 and 999 you do not explicitly say tha the SR-VIO
> is sent
> to the track ingres and that the SF-VIO is sent to the track egres. Please
> add such sentences accordingly.
> 
added

   A P-DAO that creates or updates a Track Leg MUST be sent to a GUA or a ULA
   of the Ingress of the Leg; it must contain the full list of hops in the
   Leg unless the Leg is being removed.
   A P-DAO that creates a new Track Segment MUST be sent to a GUA or a ULA
   of the Segment Egress and signal the full list of hops in Segment; a
   P-DAO that updates (including deletes) a section of a Segment MUST be
   sent to the first node after the modified Segment and signal the full
   list of hops in the section starting at the node that immediately
   precedes the modified section.

> I think the description of ACK processing from 976++ fits better here
> after, given
> how its an additional detail happening after the P-DAO is processed
> according
> to the two bullet points above.
> 
> Q: If in the SF-VIO case, the egress does not install any state, then why
> send
> the P-DAO to it instead of to the pre-ultimate hop ? Explanation might be
> helpful
> in the text.

The P-DAO goes along the reverse path that the packets will follow and check the path on the way.

> 
>  1016
>  1017       In case of a forwarding error along a Projected Route, an
> ICMP error
> 
> Arre we talking about the forwaring of data packets that are following a
> projected route,
> or processing error of the P-DAO ? Please be specific. An example of a
> forwarding
> error would help.

  In case of a failure forwarding a packet along a P-Route, e.g., the
  next hop is unreachable, the node that discovers the fault MUST send
  an ICMPv6 Error message [RFC4443] to the Root, with a new Code "Error
  in P-Route" (See Section 10.13). The message MUST be throttled in
  case of consecutive occurrences.

> I suspect it is the data packets. Is there also any other error for
> processing of
> the P-DAO other than missing ACK ? Aka: wha exactly happens in the
> duplicate address
> in VIO If lets say the root was too stsupid to figure it out ?

Per your earlier point I added

  A node that processes a VIO MAY verify whether one of these
  conditions happen, and when so, it MUST ignore the P-DAO and reject
  it with a RPL Rejection Status of "Error in VIO" in the DAO-ACK, see
  Section 10.14.

> 
>  1018       is sent to the Root with a new Code "Error in Projected
> Route" (See
>  1019       Section 11.13).  The Root can then modify or remove the
> Projected
>  1020       Route.  The "Error in Projected Route" message has the same
> format as
>  1021       the "Destination Unreachable Message", as specified in RFC
> 4443
>  1022       [RFC4443].
>  1023
>  1024       The portion of the invoking packet that is sent back in the
> ICMP
>  1025       message SHOULD record at least up to the RH if one is
> present, and
>  1026       this hop of the RH SHOULD be consumed by this node so that
> the
>           ^^^^                                     ^^^^
> which hop ? The hop that has problems and wants to generate the ICMP error?

Clarified as:

  In case of a failure forwarding a packet along a P-Route, e.g., the
  next hop is unreachable, the node that discovers the fault MUST send
  an ICMPv6 Error message [RFC4443] to the Root, with a new Code "Error
  in P-Route" (See Section 10.13).  The Root can then repair by
  updating the broken Segment and/or Tracks, and in the case of a
  broken Segment, remove the leftover sections of the segment using SM-
  VIOs with a lifetime of 0 indicating the section ot one or more nodes
  being removed (See Section 6.5).
> 
>  1027       destination in the IPv6 header is the next hop that this
> node could
>  1028       not reach.  if a 6LoWPAN Routing Header (6LoRH) [RFC8138] is
> used to
> 
> So ultimately, you want the root to be able to figure out from the ICMP,
> which nodeA can not reach a next-hop nodeB, right ? WOuld be good to say
> that,
> but the description of how to achieve this by messing with the ICMP error
> reported original packet stub still escapes me. Is that "consumed by"
> procedure
> already done some place else ? If so, hen a reference to prior way how this
> is
> done would be good. If not, then elaborating a bit more to make it easier
> to
> understand would help.

I added 

  In case of a failure forwarding a packet along a Segment, e.g., the
  next hop is unreachable, the node that discovers the fault MUST send
  an ICMPv6 Error message [RFC4443] to the Root, with a new Code "Error
  in P-Route" (See Section 10.13).  The Root can then repair by
  updating the broken Segment and/or Tracks, and in the case of a
  broken Segment, remove the leftover sections of the segment using SM-
  VIOs with a lifetime of 0 indicating the section ot one or more nodes
  being removed (See Section 6.5).

  In case of a forwarding error along a Source Route path, the node
  that fails to forward SHOULD send an ICMP error with a code "Error in
  Source Routing Header" back to the source of the packet, as described
  in section 11.2.2.3. of [RPL].  Upon this message, the encapsulating
  node SHOULD stop using the source route path for a period of time and
  it SHOULD send an ICMP message with a Code "Error in P-Route" to the
  Root.  Failure to follow these steps may result in packet loss and
  wasted resources along the source route path that is broken.

  Either way, the ICMP message MUST be throttled in case of consecutive
  occurrences.  It MUST be sourced at the ULA or a GUA that is used in
  this Track for the source node, so the Root can establish where the
  error happened.

> 
>  1028       not reach.  if a 6LoWPAN Routing Header (6LoRH) [RFC8138] is
> used to
>  1029       carry the IPv6 routing information in the outer header then
> that
>  1030       whole 6LoRH information SHOULD be present in the ICMP
> message.
> 
> How would a root then figure out nodeA/nodeB ?

Added text above

> 
>  1031
>  1032       The sender and exact operation depend on the Mode and is
> described in
>  1033       Section 7.3.2 and Section 7.3.1 respectively.
>  1034
>  1035    7.1.  Requesting a Track
>  1036
>  1037       A Node is free to ask the Root for a new Track for which it
> will be
>  1038       Ingress at any time.  This is done with a PDR message, that
> indicates
> 
> for which it wants to be ingres ? E.g.: no idea what "will be at any time"
> means.

added

     This specification introduces the PDR message, used by an LLN node to
     request the formation of a new Track for which this node is Ingress.

> 
>  1039       the desired TrackID and the duration for which the Track
> should be
>  1040       established.  Upon a PDR, the Root MAY install the necessary
> 
> An example of why the node would generate it, and where it has the TrackID
> from
> would help here.
> 
>  1041       Segments, in which case it answers with a PDR-ACK indicating
> the
>  1042       granted Track Lifetime.  All the Segments MUST be of a same
> mode,
>  1043       either Storing or Non-Storing.  All the Segments MUST be
> created with
>  1044       the same TrackID and the same DODAGID signaled in the P-DAO.
>  1045
>  1046       The Root is free to design the Track as it wishes, and to
> change the
>  1047       Segments overtime to serve the Track as needed, without
> notifying the
>  1048       resquesting Node.  The Segment Lifetime in the P-DAO
> messages does
>             ^
> Presumably, a requesting node would likely also be the ingres of one P-DAO
> of
> the track, right ? But establishment of the track via P-DAO is only related
> to
> the PDR/PDR-ACK in the root ?

This is not specified. The Root could also decide to form a track based on application request, but then there must be a method to avoid collision in the trackID name space.

Added

     This specification introduces the PDR message, used by an LLN node to
     request the formation of a new Track for which this node is Ingress.
     Note that the namespace for the TrackID is owned by the Ingress node,
     and in the absence of a PDR, there must be some procedure for the Root to
     assign TrackIDs in that namespace while avoiding collisions.

> Might be useful to have some place a higher level description of
> "initiation" models,
> and point to it. Aka: the way i figure, initiation can come from PCE and go
> through root,
> no other nodes involved. Or it comes somehow via PDR initiaor... but
> how/why.
> 
> Also "free to " is a bit of a misnomer, right ? Aka: If the root is free,
> then it

Removed free

> should also be able to notify the requesting node, but there really is no
> other
> notification than acknowledging in PDR-ACK acceptance to take care of the
> request
> or to indicate failure of it. Aka, write maybe: "The root designs the Track
> as it wishes....
> there are no notifications to the requesting node when changing the track".

Done
   
> 
>  1049       not need to be aligned to the Requested Lifetime in the PDR,
> or
>  1050       between P-DAO messages for different Segments.  The Root may
> use
>  1051       shorter lifetimes for the Segments and renew them faster
> than the
>  1052       Track is, or longer lifetimes in which case it will need to
> tear down
>  1053       the Segments if the Track is not renewed.
>  1054
>  1055       When the Track Lifetime that was returned in the PDR-ACK is
> close to
>  1056       elapse, the resquesting Node needs to resend a PDR using the
> TrackID
>  1057       in the PDR-ACK to extend the lifetime of the Track, else the
> Track
>  1058       will time out and the Root will tear down the whole
> structure.
> 
> How close ? Any guidance ?

Depends on the LLN
Added 

         - vs. the trip time from the node to the Root
> 
>  1068
>  1069       If the Track fails and cannot be restored, the Root notifies
> the
>  1070       resquesting Node asynchronously with a PDR-ACK with a Track
> Lifetime
>             ^
>  1071       of 0, indicating that the Track has failed, and a PDR-ACK
> Status
>  1072       indicating the reason of the fault.
>  1073
>  1074    7.2.  Identifying a Track
>  1075
>  1076       RPL defines the concept of an Instance to signal an
> individual
>  1077       routing topology but does not have a concept of an
> administrative
>  1078       distance, which exists in certain proprietary
> implementations to sort
>  1079       out conflicts between multiple sources of routing
> information within
>  1080       one routing topology.
>  1081
>  1082       This draft leverages the RPL Instance model as follows:
> 
> ... to sort out conflicts between multiple sources of routing information
> ??
> aka: tie the text together, the way it is written it is confusing.
> 
> It also seems as if it would be better to high level answer how P-DAO
> avoids
> the need for administrative distances here, instead of letting readers try
> to figure it
> out from all the details of the following bullet points, because those
> bullet points
> explain a lot more, so the whole administative distance stuff will get
> lost.

Not critical for understanding the section
Changed to

    RPL defines the concept of an Instance to signal an individual
    routing topology, and multiple topologies can coexist in the same network.
    The RPLInstanceID is tagged in the RPI of every packet to signal which
    topology the packet actually follows.

> 
> The way i see it is that P-DAO do not require administrative distance
> because
> 
> a) They will be preferred because of prefixlength vs. default route to root
> when
>   used within pre-existing RPLinstance.
> b) They can be explicitly indicated through their own RPLinstance called
> TrackID
>   in the RPL packet header, therefore not colliding with routes from pre-
> existing
>   RPLinstances.

Ack; the problem will arise when 2 tracks get to the same destination, which we suggest to avoid. And in storing mode since more specific route exist in the dodag which may conflict.

>  1083
>  1084       *  The Root MAY use P-DAO messages to add better routes in
> the main
>  1085          (Global) Instance in conformance with the routing
> objectives in
>  1086          that Instance.  To achieve this, the Root MAY install an
> Storing-
> 
> What actually happens if those P-DAO routes are not in conformance with the
> routing
> objectives of that instance ? Anything worse than that the routing
> objective does
> not well express what the routes are doing ?

Could write a book. Same as SDN...

> 
> I am asking, because i would assume that the routing objectives likely will
> never
> be able to express all the characteristics that may come out of a bunch of
> Tracks.
> Therefore i fear that this conformance sentence is somewhat career limiting
> to
> P-DAOs.
> 
Yes, a real complex problem. How do you map appl level SLA to network concepts?

> But i am not sure about the overall framework. Is there some form of policy
> framework,
> whereby traffic may choose one out of multiple instances, and what you
> really want
> to say is that Tracks should be assigned to the track that best matches
> what the
> Track attempts to improve upon so that traffic choosing one out of multiple
> insances can continue to do so, and track will improve the desired outcome
> ?
> 

All that policy game is out of scope but has to be considered soon after.
For now we build a shorted path for a more specific route, without characterization of how good it is.

> Aka: Motivating this setnence, maybe in the way i am imagining it (if
> correct) might
> help. And it might also better capture the goal of choosing a particular
> instance
> for a track than referring to the formal routing objectives instead of
> maybe
> the informal intent of an instance, which may be constituted from the
> routing
> objectives of DAOs and the traffic engineered objectives of P-DAOs.

Out of scope, keep for next

> 
>  1086          that Instance.  To achieve this, the Root MAY install an
> Storing-
>                                                                     ^ just
> a - storing is spoken with consonant first.
>  1087          Mode P-Route along a path down the main Non-Storing Mode
> DODAG.
>  1088          This enables a loose source routing and reduces the size
> of the
>  1089          Routing Header, see Appendix A.1.
>  1090
>  1091          When adding an Storing-Mode P-Route to the main RPL
> Instance, the
>                           ^ just a
>  1092          Root MUST set the RPLInstanceID field of the P-DAO
> message (see
>  1093          section 6.4.1. of [RPL]) to the RPLInstanceID of the main
> DODAG,
>  1094          and MUST NOT use the DODAGID field.  A Projected Route
> provides a
> 
> Whats the significance of the DODAGID here ? What would happen if it was
> set ?

a) see RPL and all the discussion on tuple identification of Track. Adding it would waste bandwidth. That is consistent with RPL.

> 
>  1095          longer match to the Target Address than the default route
> via the
>  1096          Root, so it is preferred.
>  1097
>  1098          Once the Projected Route is installed, the intermediate
> nodes
>  1099          listed in the SF-VIO after first one (i.e.  The ingress)
> can be
>  1100          elided from the RH in packets sent along the Segment
> signaled in
>  1101          the P-DAO.  The resulting loose source routing header
> indicates
>  1102          (one of) the Target(s) as the next entry after the
> ingress.
>  1103
>  1104       *  The Root MAY also use P-DAO messages to install a
> specific (say,
>  1105          Traffic Engineered) path as a Serial or as a Complex
> Track, to a
> 
> Serial Track and Complex Track are not defined/explained yet. See comments
> earlier that you need to start off with pictured examples from which to
> much more easily define such terms.
> 

Now it is, in great details, in section 3

>  1106          particular endpoint that is the Track Egress.  In that
> case, the
>  1107          Root MUST install a Local RPL Instance (see section 5 of
> [RPL]),
>  1108          and the Local RPLInstanceID is called TrackID.
> 
> Do i need a separate RPL instance to loosely tie together multiple
> sequential
> SF-VIO ?  Lets say i have 20 hops in stateless. I figure i have 3 segments
> of
> 5 hops each that ae often used. So i establish separate tracks for each of
> them, and then my routing headers in the future would be able to be reduced
> by 3 * 5 = 15 hops, aka: having 3 loose hops tied together by 5 strict
> hops.

Sure, that's a main goal of this spec. see A.1.  Loose Source Routing. 

This is now clarified in section 3

 A P-Route may be installed in either Storing and Non-Storing Mode,
  potentially resulting in hybrid situations where the Mode of the P-
  Route is different from that of the main RPL Instance.  P-Routes can
  be used as stand-alone segments to reduce the size of the source
  routing headers with loose source routing operations down the main
  RPL DODAG.  P-Routes can also be combined with other P-Routes to form
  a more complex forwarding graph called a Track.

> 
>  1109
>  1110          In that case, the TrackID MUST be unique for the Global
> Unique
>  1111          IPv6 Address (GUA) or Unique-Local Address (ULA) of the
> Track
> 
> You where missing those expansions in before, so here they would now be
> redundant
> when you fix them up in before.

Resolved

> 
>  1112          Ingress that serves as DODAGID for the Track.  The Track
> Ingress
>  1113          owns the namespace of its TrackIDs, so it can pick any
> unused
>  1114          value to request a new Track with a PDR.  The Root is
> aware of all
> 
> The root of the main DODAG, or main RPL instance or the like ? Aka: There
> may be
> multiple roots for different instances/dodags, right ?

Right. Changed to

    In that case, the TrackID MUST be unique for the IPv6 ULA or GUA of the
    Track Ingress that serves as DODAGID for the Track. The Track Ingress owns
    the namespace of its TrackIDs, so it can pick any unused value to request a
    new Track with a PDR. In a particular deployment where PDR are not used,
    the namespace can be delegated to the main Root, which can  assign the
    TrackIDs for the Tracks it creates without collision.

> 
>  1115          the active Tracks, so it can also pick any unused value
> to form
>  1116          Tracks without a PDR.  To avoid a collision of the Root
> and the
>  1125          Track Ingress picking the same value at the same time, it
> is
>  1126          RECOMMENDED that the Track Ingress starts allocating the
> ID value
>  1127          of the Local RPLInstanceID (see section 5.1. of [RPL])
> used as
>  1128          TrackIDs with the value 0 incrementing, while the Root
> starts with
>  1129          63 decrementing.
> 
>  1131          This way, a Track is uniquely identified by the tuple
> (DODAGID,
>  1132          TrackID) where the TrackID is always represented with the
> D flag
>  1133          set to 0.
> 
> Explain where the D flag is, first time i think its used in the text.

Added 

     This way, a Track is uniquely identified by the tuple (DODAGID,
     TrackID) where the TrackID is always represented with the D flag
     set to 0 (see also section 5.1. of [RPL]), indicating when used in
     an RPI that the source address of the IPv6 packet signals the
     DODAGID.

> 
> I think i may be confused about the namespace and its limitations or at
> least
> not well educaed by the text ;-)
> 
> So the track ingres IPv6 serves as DODAGID. So every node can have up to 64
> - N Tracks,
> where N is the number of non-rack RPL instances for which the node is the
> root ?

Yes

Added 

  Section 5.1. of [RPL] describes the RPL Instance ID and its encoding.
  There can be up to 128 global RPL Instances, for which there can be
  one or more DODAGs, and there can be 64 local RPL Instances, with a
  namespace that is indexed by a DODAGID, where the DODAGID is a Unique
  Local Address (ULA) or a Global Unicast Address (GUA) of the Root of
  the DODAG.

> 
> Would be good to write this out explicitly (or accordingly corrected if
> wrong).
> 
>  1135          The Track Egress Address and the TrackID MUST be signaled
> in the
>  1136          P-DAO message as shown in Figure 1.
>  1137
>  1138    7.3.  Installing a Track
>  1139
>  1140       A Storing-Mode P-DAO contains an SF-VIO that signals the
> strict
>  1141       sequence of consecutive nodes to form a segment between a
> segment
>  1142       ingress and a segment egress (both included).  It installs a
> route of
>  1143       a higher precedence along the segment towards the Targets
> indicated
>  1144       in the Target Options.  The segment is included in a DODAG
> indicated
> 
> Ok, now i am getting confused again by this text.
> 
> "higher precendence - than what ? Also, is there a definition of precedence
> in the
> context of RPL ? If so, reference to definition would be good.

Added 

  A route along a Track for which the TrackID is not the RPLInstanceID
  of the Main DODAG MUST be installed with a higher precedence than the
  routes along the Main DODAG, meaning that:

  *  Longest match MUST be the prime comparison for routing.

  *  In case of equal length match, the route along the Track MUST be
     preferred vs. the one along the Main DODAG.

  *  There SHOULD NOT be 2 different Tracks leading to the same Target
     from same Ingress node, unless there's a policy for selecting
     which packets use which Track; such policy is out of scope.

  *  A packet that was routed along a Track MUST NOT be routed along
     the main DODAG again; if the destination is not reachable as a
     neighbor by the node where the packet exits the Track then the
     packet MUST be dropped.

> 
> Previously you only used precedence related to priority to store/maintain
> a route, which i guess is a differrent meaning of the word than here.
> 
> In 7.2, it sounded as if there would be no pre-existing route to the target
> other than through the root because of prefix-length. If that is what you
> mean
> here, and unless precedence has a well defined meaning, maybe just replace
> precedence
> with prefix-length.
> 
> It is also not quite clear to me if tracks can never be installed into
> storing-mode
> RPL instances where they could not conflict with equal-prefix-length
> "native" routes.
> AFAIK, the spec only said the main RPLinstance must be non-storing mode.
> Can there nit
> be other, storing mode RPL instances ?

You're spot on

Before your review and the possibility of a storing mode main dodag there could not be a prefix length collision. 
So the precedence was implicit due to longest match, though nothing was said about it.

I hope the above clarifies it all.

> 
> But of course, i am likely confused...

Certainly not

> 
>  1144       in the Target Options.  The segment is included in a DODAG
> indicated
>  1145       by the P-DAO Base Object, that may be the one formed by the
> main RPL
>  1146       Instance, or a Track associated with a local RPL Instance.
> A Track
>  1147       Egress is signaled as a Target in the P-DAO, and as the last
> entry is
>  1148       an SF-VIO of a last segment towards that Egress.
>  1149
>  1150       A Non-Storing-Mode P-DAO signals a strict or loose sequence
> of nodes
>                                   ^ contains an SR-VIO to...
>  1151       between the Track Ingress (excluded) and a Track Egress
> (included).
>  1152       It installs a source-routed path of a higher precedence
> within the
>                                                        ^^^^^^^^^^ same
> consideration as above

I agree that precedence was used with 2 meanings. 

The other text is changed to 

   Upon receiving a propagated DAO, all except the Egress router MUST install a
   route towards the DAO Target(s) via their successor in the SM-VIO. A router
   that cannot store the routes to all the Targets in a P-DAO MUST reject the
   P-DAO by sending a DAO-ACK to the Root with a Rejection Status of "Out of
   Resources" as opposed to forwarding the DAO to its predecessor in the list.
   The router MAY install additional routes towards the VIA Addresses that are
   the SM-VIO after self, if any, but in case of a conflict or a lack of
   resource, the route(s) to the Target(s) are the ones that must be installed
   in priority.

>  1153       Track indicated by the P-DAO Base Object, towards the
> Targets
>  1154       indicated in the Target Options.  The source-routed path
> requires a
>  1155       Source-Routing header which implies an encapsulation to add
> the SRH
> 
> ^^^^^^^^^^^
> 
> Same consideration as much earlier, e.g.: encapsulate into new IPv4 with
> SRH ?!

IPv6 but yes. Added Ip in IP. 

>  1156       to an existing packet.
>  1157
>  1158       The next entry in the sequence must be either a neighbor of
> the
>  1159       previous entry, or reachable as a Target via another
> Projected Route,
>  1160       either Storing or Non-Storing.  If it is reachable over a
> Storing
>  1161       Mode Projected Route, the next entry in the loose sequence
> is the
>  1162       Target of a previous segment and the ingress of a next
> segment; the
>  1163       segments are associated with the same Track, which avoids
> the need of
>  1164       an encapsulation.  Conversely, if it is reachable over a
> Non-Storing
>  1165       Mode Projected Route, the next loose source routed hop of
> the inner
>  1166       Track is a Target of a previous Track and the ingress of a
> next
>  1167       Track, which requires a de- and a re-encapsulation.
> 
> This is a game of blindfold chess. Pictured examples of the two cases
> described
> would help a lot to understand and verify the points made/claimed here.

Pointed on the examples, e.g.,

  If the next entry in the loose sequence is reachable over a Storing
  Mode P-Route, it MUST be the Target of a Segment and the Ingress of a
  next segment, both already setup; the segments are associated with
  the same Track, which avoids the need of an additional encapsulation.
  For instance, in Section 3.4.1.3, Segments A==>B-to-C and
  C==>D==>E-to-F must be installed with Storing-Mode P-DAO messages 1
  and 2 before the Track A-->C-->E-to-F that joins them can be
  installed with Non-Storing-Mode P-DAO 3.

  Conversely, if it is reachable over a Non-Storing Mode P-Route, the
  next loose source-routed hop of the inner Track is a Target of a
  previously installed Track and the Ingress of a next Track, which
  requires a de- and a re-encapsulation when switching the outer Tracks
  that join the loose hops.  This is examplified in Section 3.4.2.3
  where Non-Storing Mode P-DAO 1 and 2 install strict Tracks that Non-
  Storing Mode P-DAO 3 joins as a super Track.  In such a case, packets
  are subject to double IP-in-IP encapsulation.

> 
>  1181       A Serial Track is installed by a single Projected Routes
> that signals
>                                                                  ^
>  1182       the sequence of consecutive nodes, either in Storing or Non-
> Storing
>  1183       Mode.  If can be a loose Non-Storing Mode Projected Route,
> in which
>                   ^
>  1184       case the next loose entry must recursively be reached over a
> Serial
>  1185       Track.
> 
> Please check whats standard *-Storing-Mode or *-Storing Mode - and make
> text
> consistent.
> 

Removed the - 

>  1186
>  1187       A Complex Track can be installed as a collection of
> Projected Routes
>  1188       with the same DODAGID and Track ID.  The Ingress of a Non-
> Storing
> 
> If this is turned around, does it become the still missing definition of a
> complex track ?

Now there's a whole section 

3.5.  Complex Tracks

Before that we have

  A Serial Track comprises provides only one path between Ingress and
  Egress.  It comprises at most one Leg. A Stand-Alone Segment defines
  implicitly a Serial Track from its Ingress to Egress.

  A complex Track forms a graph that provides a collection of potential
  paths to provide redundancy for the packets, either as a collection
  of Legs that may be parallel or cross at certain points, or as a more
  generic DODAG that is rooted at the Ingress Node.

> 
> Multiple projected routes can be installed into a single (DODAGID,TrackID).
> This leads to Sequential or Complex Tracks ??

Sequential if sequential else complex... I hope the definition above helps.

> 
>  1188       with the same DODAGID and Track ID.  The Ingress of a Non-
> Storing
>  1189       Mode Projected Route must be the owner of the DODAGID.  The
> Ingress
>  1190       of a Storing Mode Projected Route must be either the owner
> of the
>  1191       DODAGID, or the egress of a preceding Storing Mode Projected
> Route in
>  1192       the same Track.  In the latter case, the Targets of the
> Projected
>  1193       Route must be Targets of the preceding Projected Route to
> ensure that
>  1194       they are visible from the track Ingress.
>  1195
>  1196    7.3.1.  Storing-Mode P-Route
> 
> Please go back through the text and replace all instances of Projected
> Route
> after the first occurance with P-Route. You use both terms randomnly
> interchanged,
> which is confusing. Better to stick to the abbreviation only.

OK; I used what seemed to ring in my ear.

> 
> I also observe, that i would be much less confused about the text if this
> high-level
> explanation was preceeding the detailed explanation earlier in the
> beginning of section 7.
> 
> Please think how to resort the text of section 7 so it starts with the
> pictures
> and high-level explanation and then goes into the more formal detail spec.

Done

> 
>  1198       Profile 1 extends RPL operation in a Non-Storing Mode
> network with
> 
> Without further explanation, the first unexplained use of Profile is
> derailing and
> confusing. Suggest to just delete the mentioning of profiles and just
> define them
> in Section 8.

Done

>  1199       Storing-Mode Projected Routes that install segments along
> the main
>  1200       DODAG and enable to loose source routing between the Root
> and the
>  1201       targets.
>  1202
>  1203    7.3.1.1.  Installing a Storing-Mode P-Route
>  1204
>  1205       As illustrated in Figure 9, a P-DAO that carries a SF-VIO
> enables the
>  1206       Root to install a stateful route towards a collection of
> Targets
>  1207       along a Segment between a Track Ingress and a Track Egress
> using a
>  1208       projected DAO Message.
>  1209
>  1210               ------+---------
>  1211                     |          Internet
>  1212                     |
>  1213                  +-----+
>  1214                  |     | Border Router
>  1215                  |     |  (RPL Root)
>  1216                  +-----+                      |     ^
> |
>  1217                     |                         | DAO | ACK
> |
>  1218               o    o   o    o                 |     |
> |
>  1219           o o   o  o   o  o  o o   o          |  ^       |
> Projected    .
>  1220          o  o o  o o    o\  o   o  o  o       |  | DAO   | Route
> .
>  1221          o   o    o  o    \o--o    o  o  o    | ^        |
> .
>  1222         o  o   o  o   o        \o   o o       v | DAO    v
> .
>  1223         o          o   LLN   T1   T2     o
> |
>  1224             o o   o        o     o              Loose Source Route
> Path |
>  1225          o       o      o    o                 From Root To
> Destination v
>  1226
>  1227                            Figure 9: Projecting a route
> 
> 
> Do you think you can visualise an example P-route into the picture ?
> I started a bit of bit of it above..
> 
>  1228
>  1237       In order to install the relevant routing state along the
> Segment ,
>  1238       the Root sends a unicast P-DAO message to the Track Egress
> router of
>  1239       the routing Segment that is being installed.  The P-DAO
> message
>  1240       contains a SF-VIO with the direct sequence of Via Addresses.
> The SF-
>                                      ^^^^^^ strict

Done

> 
>  1241       VIO follows one or more RTOs indicating the Targets to which
> the
>  1242       Track leads.  The SF-VIO contains a Segment Lifetime for
> which the
> 
> In this explanation it would be helpful to explain conditions for targets,
> e.g.:
> Target must either be direc neighbor of the P-Route egres, or be Target of
> some
> P-Route2 with the P-Route egres as its ingres. Right ?

This is true and discussed prior 

> 
>  1243       state is to be maintained.
>  1244
>  1245       The Root sends the P-DAO directly to the egress node of the
> Segment.
>  1246       In that P-DAO, the destination IP address matches the last
> Via
>  1247       Address in the SF-VIO.  This is how the egress recognizes
> its role.
>  1248       In a similar fashion, the ingress node recognizes its role
> as it
>  1249       matches first Via Address in the SF-VIO.
>  1250
>  1251       The Egress node of the Segment is the only node in the path
> that does
>  1252       not install a route in response to the P-DAO; it is expected
> to be
>  1253       already able to route to the Target(s) on its own.  If one
> of the
>  1254       Targets is not known, the node MUST answer to the Root with
> a
>  1255       negative DAO-ACK listing the Target(s) that could not be
> located
>  1256       (suggested status 10 to be confirmed by IANA).
> 
> So complex/sequential Tracks need to be established from destination to
> source.
> What happens with a track when th egress looses routing information for a
> destination ?
> (if thats answere elsehwhere, a pointer to that section would be helpfull
> here).

If that's the forwarding error there's effectively text in the forwarding section.
Should not duplicate should we?

> 
>  1258       If the egress node can reach all the Targets, then it
> forwards the
>  1259       P-DAO with unchanged content to its loose predecessor in the
> Segment
> 
> Shouldn't that be strict instead of loose ? If it was loose, how would that
> loose
> predecessor know how to send the data packets to the successor without
> adding
> another encap/RH if the underlying main RPL instance was NSM ?

Correct, removed loose

> 
>  1260       as indicated in the list of Via Information options, and
> recursively
>  1261       the message is propagated unchanged along the sequence of
> routers
>  1262       indicated in the P-DAO, but in the reverse order, from
> egress to
>  1263       ingress.
>  1264
>  1265       The address of the predecessor to be used as destination of
> the
>  1266       propagated DAO message is found in the Via Address the
> precedes the
>  1267       one that contain the address of the propagating node, which
> is used
>  1268       as source of the message.
>  1269
>  1270       Upon receiving a propagated DAO, all except the Egress
> Router MUST
>  1271       install a route towards the DAO Target(s) via their
> successor in the
>  1272       SF-VIO.  The router MAY install additional routes towards
> the VIA
>  1273       Addresses that are the SF-VIO after the next one, if any,
> but in case
>  1274       of a conflict or a lack of resource, the route(s) to the
> Target(s)
>  1275       have precedence.
>  1276
>  1277       If a router cannot reach its predecessor in the SF-VIO, the
> router
>  1278       MUST answer to the Root with a negative DAO-ACK indicating
> the
>  1279       successor that is unreachable (suggested status 11 to be
> confirmed by
>  1280       IANA).
>  1281
>  1282       The process continues till the P-DAO is propagated to
> ingress router
>                                 ^until
>  1283       of the Segment, which answers with a DAO-ACK to the Root.
> The Root
>  1284       always expects a DAO-ACK, either from the Track Ingress with
> a
>  1293       positive status or from any node along the segment with a
> negative
>  1294       status.  If the DAO-ACK is not received, the Root may retry
> the DAO
>  1295       with the same TID, or tear down the route.
>  1296
>  1297    7.3.1.2.  Maintaining and Tearing Down a Storing-Mode P-Route
>  1298
>  1299       A Segment Lifetime of 0 in a VIO is used to clean up the
> state.  The
>                                                                    ^
>                                                                    P-route
> state
>  1300       P-DAO is forwarded as described above, but the DAO is
> interpreted as
>  1301       a No-Path DAO and results in cleaning up existing state as
> opposed to
>  1302       refreshing an existing one or installing a new one.
>  1303
>  1304       Note that the continuity of the segment may be broken; this
> happens
>  1305       if the bidirectional connectivity between contiguous hops of
> the
>  1306       segment is lost.  In that case the Root needs to send the
> projected
>  1307       DAO to one or more intermediate node(s) as opposed to the
> egress
>  1308       node, indicating a portion of segment that is being replaced
> or
>  1309       cleaned up.  At the extreme, the P-DAO updates only one
> node, in
>  1310       which case it contains only one VIO.

> 
> Hmm... how does this work given how the egres is meant to not install route
> state,
> so if you want to fix up some intermediate segments, then the egres of that
> fixup VIO, which is not the final egres of the full VIO  would from the
> prior
> explanation have to invoke its function of checking the targets
> reachability,
> but not to establish storing mode state. Some more explanation here might
> be
> helpful. I think his can work, but i think the egres of a fixup VIO needs
> to
> be careful and behave differently to a non-fixup actual track egres.

That's the case where the P-DAO removes a state (as opposed to install) so yes that can be only one node, say being replaced by another.

>  1311
>  1312       In case of a forwarding error along an Storing-Mode P-Route,
> the node
>  1313       that fails to forward SHOULD send an ICMP error with a code
> "Error in
>  1314       Projected Route" to the Root.  Failure to do so may result
> in packet
>  1315       loss and wasted resources along the Projected Route that is
> broken.
>  1316
>  1317    7.3.2.  Non-Storing-Mode P-Route
>  1318
>  1319       As illustrated in Figure 10, a P-DAO that carries an SR-VIO
> enables
>  1320       the Root to install a source-routed path from a Track
> Ingress towards
>  1321       a Target along the main DODAG.
>  1349
>  1349                  ------+---------
>  1350                        |          Internet
>  1351                        |
>  1352                     +-----+
>  1353                     |     | Border Router
>  1354                     |     |  (RPL Root)
>  1355                     +-----+                    |  P  ^ ACK
>  1356                        |        Track          | DAO |
>  1357                  o    o   o  o  Ingress X      V     |   X
>  1358              o o   o  o   o  o     o   X   o             X Source
>  1359             o  o o  o o    o   o  o    X  o  o           X Routed
>  1360             o   o    °  o     o   o   o X     o          X Segment
>  1361            o  o   o  o   o  o    o  o     X Track        X
>  1362               o  o  o  o             o     Egress
>  1363
>  1364              o       o               o    o
>  1365            o          o             o     o
>  1366                                          destination
>  1367
>  1368                              LLN
>  1369
>  1370                     Figure 10: Projecting a Non-Storing Route
> 
> So there is some attempt here to visualize the P-Route, but it mentions
> destination
> and not target, so i am again confused about the difference of those
> terms...
> Would be good to show Target(s) in the picture.

Done

> 
>  1372       When forwarding a packet to a destination for which the
> router
> 
> which router ? The Track ingres ?

Changed to

  When forwarding a packet to a destination for which a router determines
  that routing happens via a Track for which it is Ingress, the router must
  encapsulated the packet using IP-in-IP to add the Source Routing Header 
  with the final destination set to the Track Egress.

>  1373       determines that routing happens via a Track Target, the
> router
> 
> "via a track target" sounds as if destination is different from trac
> target.
> What is the routing information from which that router can determine that
> a destination is reachable via a target ? The P-Route only signals targets,
> right ?

Right: "target" was extraneous

> 
>  1374       inserts the Source Routing Header in the packet with the
> final
>  1375       destination at the Track Egress.
> 
> encapsulates the packet with a new IPv6/SRH header ?
> 
> Now it sounds as if final destination is the same as track egres, so not
> even target...
> even more confused now.

The outer destination (of IP in IP) is the Track egress. Which decapsulates and forwards to the Target.

>  1376
>  1377       In order to signal the Segment, the router encapsulates the
> packet
>  1378       with an IP-in-IP header and a Routing Header as follows:
> 
> It seems as if we're not discussing installation of non-storing P-Routes at
> all,
> but immediately skip to data-plane for non-storing P-Routes ?
> 

You right, I move all that text to a section " Encapsulating and Forwarding Along a Track "
After installing a Track.

> I would suggest to have even if a very short section for installation of
> the non-storing
> P-Route in one subsection and then aother subsection for the data-plane.
> 
> The storing mode text above does not seem to have a dedicated data-plane
> subsecton too,
> so quite inconsistent...

Done

> 
>  1379
>  1380       *  In the uncompressed form the source of the packet is this
> router,
> 
> this router...
> 

Aka self, doesn't it read ok?

*     In the uncompressed form the source of the packet is the address
     that this router uses as DODAGID for the Track, the destination is
     the first Via Address in the NSM-VIO, and the RH is a Source
     Routing Header (SRH) [RFC6554] that contains the list of the
     remaining Via Addresses terminating by the Track Egress.

>  1381          the destination is the first Via Address in the SR-VIO,
> and the RH
>  1382          is a Source Routing Header (SRH) [RFC6554] that contains
> the list
>  1383          of the remaining Via Addresses terminating by the Track
> Egress.
>  1384
>  1385       *  The preferred alternate in a network where 6LoWPAN Header
>  1386          Compression [RFC6282] is used is to leverage "IPv6 over
> Low-Power
>  1387          Wireless Personal Area Network (6LoWPAN) Paging Dispatch"
>  1388          [RFC8025] to compress the RPL artifacts as indicated in
> [RFC8138].
>  1389
>  1390          In that case, the source routed header is the exact copy
> of the
>  1391          (chain of) SRH-6LoRH found in the SR-VIO, also
> terminating by the
>  1392          Track Egress.  The RPI-6LoRH is appended next, followed
> by an IP-
>  1393          in-IP 6LoRH Header that indicates the Ingress Router in
> the
>  1394          Encapsulator Address field, see as a similar case Figure
> 20 of
>  1395          [TURN-ON_RFC8138].
> 
> I will trust this is correct, as i can not verify this quickly due to my
> lack of
> this encap details.

Now we have 

 6.7.  Compression of the RPL Artifacts

> 
>  1404
>  1405       In the case of a loose source-routed path, there MUST be
> either a
>  1406       segment for the same Track to the loose next hop, on which
> case the
>  1407       packet is forwarded to the next hop along that segment, a
> common
>  1408       neighbor with the loose next hop, on which case the packet
> is
>  1409       forwarded to that neighbor, or another Track to the loose
> next hop
>  1410       for which this node or a neighbor is Ingress.  In the latter
> case,
>  1411       another encapsulation takes place and the process possibly
> recurses;
>  1412       otherwise the packet is dropped.
> 
> I can not comment on this because i am still lost as to the semantic of
> segment given
> he absence of any illustrative pictures for multi-segment Tracks so far in
> the document.

Done

> 
>  1414       In case of a forwarding error along a Source Route path, the
> node
>  1415       that fails to forward SHOULD send an ICMP error with a code
> "Error in
>  1416       Source Routing Header" back to the source of the packet, as
> described
>  1417       in section 11.2.2.3. of [RPL].  Upon this message, the
> encapsulating
>  1418       node SHOULD stop using the source route path for a period of
> time and
> 
> How long should that period be. Elaborate for how the encapsulating node
> should determine the period.

Depends heavily on deployment. RPL does not specify thos  for that reason.

Updated

"                          SHOULD stop using the source route path for a
     reasonable period of time which duration depends on the deployment "

> 
>  1419       it SHOULD send an ICMP message with a Code "Error in
> Projected Route"
>  1420       to the Root.  Failure to follow these steps may result in
> packet loss
>  1421       and wasted resources along the source route path that is
> broken.
>  1422
>  1423    7.4.  Forwarding Along a Track
> 
> In 7.3.2 you already detail parts of this. Again, i would strongly suggest
> to revert order, with this section first, before getting into the currently
> earlier stated details.

Done

> 
>  1425       This draft leverages the RPL Forwarding model follows:
>  1426
>  1427       *  In the data packets, the Track DODAGID and the TrackID
> MUST be
>  1428          respectively signaled as the IPv6 Source Address and the
>  1429          RPLInstanceID field of the RPI that MUST be placed in the
> outer
>  1430          chain of IPv6 Headers.
> 
> I think an outer chain(mail) can be found as an armor on medieval knights,
> but in our
> case i think it is the outer header of the packets chain of IPv6 headers.

😊  😊  😊 
> 
>  1432          The RPI carries a local RPLInstanceID called the TrackID,
> which,
>  1433          in association with the DODAGID, indicates the Track
> along which
>  1434          the packet is forwarded.
>  1435
>  1436          The D flag in the RPLInstanceID MUST be set to 0 to
> indicate that
>  1437          the source address in the IPv6 header is set ot the
> DODAGID, more
>                                                           ^^
>  1438          in Section 7.4.
> 
> This is section 7.4.

Fixed

> 
>  1439
>  1440       *  This draft conforms the principles of [USEofRPLinfo] with
> regards
>                                 ^ to
>  1441          to packet forwarding and encapsulation along a Track.
>  1442
Ok

>  1443          -  In that case, the Track is the DODAG, the Track
> Ingress is the
> 
> In the case where the Track is the main DODAG ... ?

Rather "With this draft,"

> 
>  1444             Root, and the Track Egress is a RAL, and neighbors of
> the Track
> 
> Is there any reason why to say RAL instead of node ? First time you use
> RAL, so wondering.

Changed to

     the Track is the DODAG, the Track Ingress is the Root, and the Track Egress
     is a RPL-Aware Node (RAN), and neighbors of the Track Egress that can be 
     reached via the Track are RPL-Unaware Leaves (RULs)

> 
>  1445             Egress that can be reached via the Track are RULs.
> The
>  1446             encapsulation rules in [USEofRPLinfo] apply.
> 
> Not sure what RFC editor says, but given how RAL/RUL are first mentioned
> here,
> even though they are listed in glossary, expanding them here can not hurt.
> 
> I think you may want to find a place early in he document where you want to
> say
> that "node" without qualification in this doc mean RAN, maybe also add to
> glossary.
> 
> It would be helpfull to clariy what i think this section wants to say
> without hoping that
> readers can / want to deduce it from [UseofRPLinfo]. How about the
> following table:
> 
>        Node role                   Supported node type
>        Track ingres                RAN
>        Track midpoint              RAN
>        Track egres                 RAN, RAL
>        Neighbor of track egres     RAN, RAL, RUL
> 
> (hope this is roughly ok, else fix up).

It is OK, feels like overkill when the point is really that the Targets are RULs for the Track perspective (as a DODAG)    .

Slightly reworded

     With this draft, the Track is a RPL DODAG. From the perspective of that
     DODAG, the Track Ingress is the Root, the Track Egress is a RPL-Aware
     6LR, and neighbors of the Track Egress that can be reached via the Track,
     but are external to it, are external destinations and treated as
     RPL-Unaware Leaves (RULs).

> 
>  1448          -  If the Track Ingress is the originator of the packet
> and the
>  1449             Track Egress is the destination of the packet, there
> is no need
>  1450             for an encapsulation.
>  1451
>  1459
>  1460
>  1461          -  So the Track Ingress must encapsulate the traffic that
> it did
>  1462             not originate, and add an RPI in any fashion.
>  1463
>  1464          A packet that is being routed over the RPL Instance
> associated to
>  1465          a first Non-Storing Mode Track MAY be placed
> (encapsulated) in a
>  1466          second Track to cover one loose hop of the first Track.
> On the
> 
> I can not follow this explanation without an example picture. When reading
> it,
> the first thing that comes to mind is that it sounds as if i could have
> sequential
> second Tracks in different RPL Instances, but it is unclear whether this is
> is a predondition of this case of whether its irrelevant.

Added a reference to the non storing segment routing description 

> 
>  1466          second Track to cover one loose hop of the first Track.
> On the
>  1467          other hand, a Storing Mode Track must be strict and a
> packet that
>  1468          it placed in a Storing Mode Track MUST follow that Track
> till the
> 
> :%s/\<till\>/until/g
> Don't speak emacs.
> 

https://www.grammarly.com/blog/until-till-til/

>  1469          Track Egress.
>  1470
>  1471          When a Track Egress extracts a packet from a Track
> (decapsulates
>  1472          the packet), the Destination of the inner packet MUST be
> either
>  1473          this node or a direct neighbor, or a Target of another
> Segment of
>  1474          the same Track for which this node is ingress, otherwise
> the
>  1475          packet MUST be dropped.
> 
> .... and (see reference) ICMP must be raied according to... etc. pp ?!

Added a next para on this right after

  In case of a failure forwarding a packet along a Segment, e.g., the
  next hop is unreachable, the node that discovers the fault MUST send
  an ICMPv6 Error message [RFC4443] to the Root, with a new Code "Error
  in P-Route" (See Section 10.13).  The Root can then repair by
  updating the broken Segment and/or Tracks, and in the case of a
  broken Segment, remove the leftover sections of the segment using SM-
  VIOs with a lifetime of 0 indicating the section ot one or more nodes
  being removed (See Section 6.5).

> 
> This last paragraph should really be as early in the document as possible.

I believe it's better now

> 
>  1476
>  1477       All properties of a Track operations are inherited form the
> main RPL
>  1478       Instance that is used to install the Track.  For instance,
> the use of
>  1479       compression per [RFC8138] is determined by whether it is
> used in the
>  1480       main instance, e.g., by setting the "T" flag [TURN-
> ON_RFC8138] in the
>  1481       RPL configuration option.
> 
> Can there be multiple main instances ? If so, that should be mentioned and
> maybe an example given.

No, there's only one Main DODAG, changing all to that formulation.

> 
>  1482
>  1483    8.  Profiles
>  1484
>  1485       This document provides a set of tools that may or may not be
> needed
>  1486       by an implementation depending on the type of application it
> serves.
>  1487       This sections described profiles that can be implemented
> separately
>  1488       and can be used to discriminate what an implementation can
> and cannot
>  1489       do.
>  1490
>  1491       Profile 0  Profile 0 is the legacy support of [RPL] Non-
> Storing Mode.
>  1492          It provides the minimal common functionality that must be
>  1493          implemented as a prerequisite to all the Track-supporting
>  1494          profiles.  The other Profiles extend Profile 0 with
> selected
>  1495          capabilities that this specification introduces on top.
> 
> Is this profile a term estblished outside this document already, then
> please
> do provide reference. If it is only introuced in this document, then what
> exactly needs to be implemented for it ?
> 
> Let me guss: This is a new definition in this document, and it refers to
> nodes
> that do not implement anything from this draft, but only RPL NSM acccording
> to
> (fill in, best with section), and to deploy any of this documents
> technology,
> ALL RAL in the network MUST implement Profile 0 or better. Or maybe not...
> Maybe only RAL that are used to pass P-Route traffic MUST be Profile 0 or
> better ?!
> 
> 
> If i am guessing right than my prior paragraphs text might be a better
> starting
> point with less gussing.

Added 

   This document provides a set of tools that may or may not be needed by
   an implementation depending on the type of application it serves.
   This sections described profiles that can be implemented separately and
   can be used to discriminate what an implementation can and cannot do.
   This section describes profiles that enable to implement only a portion
   of this specification to meet a particular use case. Only Profiles 0 and 1
   are REQUIRED by all implementations; all the other profiles are OPTIONAL

Note that profile 0 is RPL NSM per RFC 6550, as is. That's why it's called zero.

> 
>  1496
>  1497       Profile 1 (Storing-Mode P-Route Segments along the main
> DODAG)  Profi
>  1498          le 1 does not create new paths; it combines Storing and
> Non-
>  1499          Storing Modes to balance the size of the routing header
> in the
>  1500          packet and the amount of state in the intermediate
> routers in a
>  1501          Non-Storing Mode RPL DODAG.
> 
> You should be able to state exactly what part of this spec a node
> MUST/SHOULD
> implement to be in compliance with Profile 1. Same is true for the other
> profiles.
> Might be difficult to structure the documents into subsections such that
> you can list those subsections that are required (incrementally) for each
> Profile,
> but otherwise its really hard to figure out if or if not an implementation
> is
> compliant with a profile.

Effectively that would be quite impractical to do. 

>  1503       Profile 2 (Non-Storing-Mode P-Route Segments along the main
> DODAG)  P
>  1504          rofile 2 extends Profile 0 with Strict Source-Routing
> Non-Storing-
> 
> Can you try to persuade tools to not split words (P rofile) ?

Sorry for that. Only the text version I dare hope

> 
>  1505          Mode Projected Routes along the main DODAG.  Profile 2
> provides
>  1506          the same capability to compress the SRH in packets down
> the main
>  1507          DODAG as Profile 1, but it require an encapsulation, in
> order to
>  1508          insert an additional SRH between the loose source routing
> hops.
>  1509
>  1517       Profile 3  Profile 3 and above build Tracks that do not
> necessarily
>  1518          follow the main DODAG.  In order to form the best path
> possible,
> 
> ^ ?
> DODAG ? "best possible" is unexplained. Might be good to give (maybe
> textual is
> sufficient) the most simple example of how an additional DODAG/TrackID
> would
> be limited if it can not support Sibling Information Option. This might be
> obvious to RPL experts of course..

Added early in the doc

   This specification expects that the RPL Main DODAG is operated in RPL
   Non-Storing Mode to sustain the exchanges with the Root. Based on its
   comprehensive knowledge of the parent-child relationship, the Root can form
   an abstracted view of the whole DODAG topology. This document adds the
   capability for nodes to advertise additional sibling information to
   complement the topological awareness of the Root to be passed on to the PCE,
   and enable the PCE to build more / better paths that traverse those siblings.
> 
>  1519          those Profiles require the support of Sibling Information
> Option
>  1520          to inform the Root of additional possible hops.  Profile
> 3 extends
>  1521          Profile 1 with additional Storing-Mode Projected Routes
> that
>  1522          install segments that do not follow the main DODAG.  If
> the
>  1523          Segment Ingress (in the SF-VIO) is the same as the IPv6
> Address of
>  1524          the Track Ingress (in the projected DAO base Object), the
> P-DAO
>  1525          creates an implicit Track between the Segment Ingress and
> the
>  1526          Segment Egress.
>  1527
>  1528       Profile 4  Profile 4 extends Profile 2 with Strict Source-
> Routing
>  1529          Non-Storing-Mode Projected Routes to form Tracks inside
> the main
>  1530          DODAG.  A Track is formed as one or more strict source
> source
>  1531          routed paths between the Root that is the Track Ingress,
> and the
>  1532          Track Egress that is the last node
> 
> Why is this Profile 4 when Profile 3 says that all further profiles
> do not necessarily follow the main DODAG ? Aka: What would not work if
> you where to make 4 become eg.: 2.5 ?

The Track is inside the DODAG but does not follow it. like east west vs north south.

> 
>  1534       Profile 5  Profile 5 Combines Profile 4 with Profile 1 and
> enables to
>  1535          loose source routing between the Ingress and the Egress
> of the
>  1536          Track.  As in Profile 1, Storing-Mode Projected Routes
> connect the
>  1537          dots in the loose source route.
>  1538
>  1539       Profile 6  Profile 6 Combines Profile 4 with Profile 2 and
> also
>  1540          enables to loose source routing between the Ingress and
> the Egress
>  1541          of the Track.
> 
> How about a table where rows are features, columns profiles and
> intersections
> checkmarks ? Right now i am lost in the details.

Hard to summarize in even less, I added text to clarify the NSM operation in the main DODAG that serves for profile 2.

  Profiles 0 to 2 operate in the Main RPL Instance and do not require
  the support of local RPL Instances or the indication of the RPL
  Instance in the data plane.  Profile 3 and above leverage Local RPL
  Instances to build arbitrary Tracks rooted at the Track Ingress and
  using its namespace for TrackID.

  Profiles 0 and 1 are REQUIRED by all implementations that may be used
  in LLNs; this enables to use Storing Mode to reduce the size of the
  Source Route Header in the most common LLN deployments.  Profile 2 is
  RECOMMENDED in high speed / wired environment to enable traffic
  Engineering and network automation.  All the other profile /
  environment combinations are OPTIONAL.
> 
>  1542
>  1543    9.  Example Track Signaling
>  1544
>  1545       The remainder of the section provides an example of how a
> Track can
>  1546       be signaled
>  1547
>  1548                                                      ===> F
>  1549                       A ===> B ===> C ===> D===> E <
>  1550                                                      ===> G
>  1551
>  1552
>  1553                             Figure 11: Reference Track
>  1554
>  1555       A is Track ingress, E is track Egress.  C is stitching
> point.  F and
> 
> First time "stitching point" is used. Can you avoid a new term here ? Else
> explain.
> 
>  1556       G are E's neighbors, "external" to the Track, and reachable
> from A
>  1557       over the Track A->E.
> 
> I guess this is a two-segment Track with one segment A-...>C and one
> segment C-...>E.
> 
> What is the "<" after E good for ?

Ascii art for a fork. Does this look better?

                                                /===> F
                  A ===> B ===> C ===> D===> E <
                                                \===> G

> 
> Why use the same symbol ===> for track and for getting from E to F, G ?
> Are F and G  Targets and/or destinations ? Please introduce those terms
> into the example.
> 
> How about indicating both the physical connectivity with "---" and the
> track with "===>"

I'd rather signal strict vs. loose with this
Added

  Conventionally we use ==> to represent a strict hop and --> for a
  loose hop.  We use -to- like in C==>D==>E-to-F to represent coma-
  separated Targets, e.g., F is a Target for Segment C==>D==>E.  In
  this example, A is Track Ingress, E is Track Egress.  C is a
  stitching point.  F and G are "external" Targets for the Track, and
  become reachable from A via the Track A-->E-to-F,G.

> 
> Also show / disinguish the two segments accordingly through better ASCII
> graphics.
> 
>  1558
>  1559       In a general manner we want:
>  1560
>  1561       *  P-DAO 1 signals C==>B==>E
> 
> I hope B should be D, else i am heck more confused.

Typo, I had already debunked that one but good catch

> 
> Please say what this signaling establishes. We learned in before that a P-
> DAO
> has not only a VIO, but also some targets. So what are the targets of this
> P-DAO ?
> 
>  1562
>  1563       *  P-DAO 2 signals A==>B==>C
> 
> Likewise

For each of the 6 cases I added something like

  In this formulation:

  *  P-DAO 1 signals C==>D==>E-to-E,F,G

  *  P-DAO 2 signals A==>B==>C-to-E,F,G

> 
>  1573       *  P-DAO 3 signals F and G via the A==>E Track
> 
> What does hthis mean ? Whats the entry, whats the exit ? whats the VIO,
> whats the
> target of P-DAO 3 ?

Which the new formulation just above hopefully explains better; I reformulate again the text as

Conventionally we use ==> to represent a strict hop and --> for a
  loose hop.  We use -to- like in C==>D==>E-to-F to represent coma-
  separated Targets, e.g., F is a Target for Segment C==>D==>E.  In
  this example, A is Track Ingress, E is Track Egress.  C is a
  stitching point.  F and G are "external" Targets for the Track, and
  become reachable from A via the Track A(ingress) to E (Egress and
  implicit Target) leading to F and G (explicit Targets).

  In a general manner the desired outcome is as follows:

  *  Targets are E, F, and G

  *  P-DAO 1 signals C==>D==>E

  *  P-DAO 2 signals A==>B==>C

  *  P-DAO 3 signals F and G via the A-->E Track

> 
>  1575       P-DAO 3 being loose, it can only be non-storing.  Note that
> since the
>  1576       Root is always the ingress of a Track, and all SR-VIOs are
> now Track,
>           ^^^^^^^^^^^^^^^^^^^^^^^^^^
> ^^^^^^^^^^^^^^^^^^^^^^^^^
> 
> Which root ? the root of the main DODA ?
> I don't parse the second ^^^^ either.

Reworded as

  Loose sequences of hops must be expressed in Non-Storing Mode, so
  P-DAO 3 contains a NSM-VIO.  With this specification, the DODAGID to
  be used by the Ingress as source address is signaled if needed in the
  DAO base object, the via list starts at the first loose hop and
  matches the source route header, and the Egress of a Non-Storing Mode
  P-DAO is an implicit Target that is not listed in the RTO.
> 
>  1577       the Root being signaled in the DAO base object can now be
> elided in
>  1578       the VIA list in SR-VIO.  This enables the construction by
> the main
>  1579       root of the RFC 8138 optimized SRH-6LoRH in the SR-VIO, to
> be placed
>  1580       as is in the packet by the Root.
> 
> I can not follow that paragraph. Would it be posible to constrain the
> example
> to not also include the SRH-6LoRH complexity, given how its an option ? One
> step at a time ?
> 
Removed from there. 

I looked again at the text based on this comment and ended as

  When the [RFC8138] compression is used, the Root of the Main DODAG
  that sets up the Track also constructs the compressed routing header
  (SRH-6LoRH) on behalf of the Track Ingress, which saves the
  complexities of optimizing the SRH-6LoRH encoding in constrained
  code.  The SRH-6LoRH is signaled in the NSM-VIO, in a fashion that it
  is ready to be placed as is in the packet encapsulation by the Track
  Ingress.

>  1581
>  1582    9.1.  Using Storing-Mode Segments
>  1583
>  1584       A==>B==>C and C==>D==>E are segments of a same Track.  Note
> that the
> 
> See above. his explanation comes too late.
> 
>  1585       storing mode signaling imposes strict continuity in a
> segment.  One
>           ^^^^^^^^^^^^^^^^^^^^^^                ^^^^^^^^^^ steering
> 
> So i am confused if actually the storing mode signaling imposes that strict
> steering. I guess if i start from the egres, and it wants to forward the P-
> DAO
> to the prior hop node in the VIO, if that prior hop is a neighbor, then it
> would send the P-DAO directly to that neighbor, but if it is not a direct
> neighbor, it would need to go through the root. I guess this is 101
> knowledge
> from RFC6550, so if i had the time to read anothe 150 pages, i might be
> able
> to deduce that, but i guess it wouldn't hurt to put the same explanation
> into
> th text, e.g.: P-DAO must be sent directly from each segment node to its
> prior segment node, and in a non-storing DODAG, this is only possible if
> they are neighbors.

All correct

Recorded to add that as

   Note that the Storing Mode signaling imposes strict continuity in a 
   segment, since the P-DAO is passed hop-by-hop, as a classical DAO is,
   along the reverse datapath that it signals. One benefit of strict
   routing is that loops are avoided along the Track.

> 
> Also:
> One of the fundamental explanations missing is the definition of the
> relationship
> between a segment and a P-Route. The way i figure, a P-DAO is (segment,
> {targets})

Yes

> and represents a set of P-routes { targetI -> segment } targetI in
> {targets}).
> 
Yes 

> And then definition of tracks as composed of sequences of P-routes
> constructed
> from likely trees of P-DAO
> 
> Or something like that...

Not really trees, more likely interlaced or parallel lines from one ingress to usually one egress.
We're missing the capability to signal north south segments that join lines.
I'm adding text to enable the signaling of a real DODAG when the nodes are not constrained and can implement more Root functionality to build their own DODAG and select  paths

7.2.  A Track as a Full DODAG

  This specification builds parallel or crossing Track Legs as opposed
  to a more complex DODAG with interconnections at any place desirable.
  The reason for that limitation is related to constrained node
  operations, and capability to store large amount of topological
  information and compute complex paths:

  *  With this specification, the node in the LLN has no topological
     awareness, and does not need to maintain dynamic information about
     the link quality and availability.

  *  The Root has a complete topological information and statistical
     metrics that allow it or its PCE to perform a global optimization
     of all Tracks in its DODAG.  Based on that information, the Root
     computes the Track Leg and predigest the source route paths.

  *  The node merely selects one of the proposed paths and applies the
     associated pre-computed routing header in the encapsulation.  This
     alleviates both the complexity of computing a path and the
     compressed form of the routing header.

  The "Reliable and Available Wireless (RAW) Architecture/Framework"
  [RAW-ARCHI] actually expects the PSE at the Track Ingress to react to
  changes in the forwarding conditions along the Track, and reroute
  packets to maintain the required degree of reliability.  To achieve
  this, the PSE need the full richness of a DODAG to form any path that
  could make meet the SLAs.

  This section specifies the additions that are needed to turn the
  Track into a full DODAG and enable the main Root to provide the
  necessary topological information to the Track Ingress.  The
  expectation is that the metrics that the PSE uses are of an order
  other than that of the PCE, because of the difference of time scale
  between routing and forwarding, mor e in [RAW-ARCHI].  It follows
  that the PSE will learn the metrics it needs from an alternate
  source, e.g., OAM frames.

  To pass the topological information to the Ingress, the Root uses a
  P-DAO messages that contains sequences of Target and Transit
  Information options that collectively represent the Track, expressed
  in the same fashion as in classical Non-Storing Mode.  The difference
  as that the Root is the source as opposed to the destination, and can
  report information on many Targets, possibly the full Track, with one
  P-DAO.

  Note that the Path Sequence and Lifetime in the TIO are selected by
  the Root, and that the Target/Transit information tupples in the
  P-DAO are not those received by the Root in the DAO messages about
  the said Targets.  The Track may follow sibling routes and does not
  need to be congruent with the Main DODAG.

> 
> 
>  1586       benefit of strict routing is that loops are avoided along
> the Track.
> 
> As long as the underlying topology does not change and direct neigbors
> start
> to becom non-diret neighbors. In which case i think the segment wold need
> to be immediately invalidated. But the question raised in before about
> ICMP errors is valid, especially when i think about path divesity
> forwarding.
> In path diversity you often do NOT want to have traffic triggered errors,
> but
> jus sit out the error, even if that trafic gets dropped somehwere along the
> path.

True

Not every time but at some point you do (when it gets permanent). 
Note that wireless links vary a lot and at RAW we use OAM to collect the statistics and report to the main Root.

> Any desirable repair to do on the topolocy could/should come ffom non-
> traffic
> trigers, such as neighbor tracking to the management plane.
> 
> Is there any way for traffic itself in the Routing information header to
> indicate
> if it does or does not want to get ICMP indications when it fails to get
> forwarded ?
> That would be ideal to distingish between non-redundant and redundant
> traffic
> ICMP reaction...

I clarify in the ICMP text that it is only upon permanent errors, but the 
Concept of permanent must remain abstract.

> 
>  1587
>  1588    9.1.1.  Stitched Segments
> 
> Sequential ?
> 
>  1589
>  1590       Storing-Mode P-DAO 1 and 2 are sent to E and C, as follows:
>  1591
>  1592                  +===============+==============+==============+
>  1593                  |     Field     | P-DAO 1 to E | P-DAO 2 to C |
>  1594                  +===============+==============+==============+
>  1595                  |      Mode     | Storing      | Storing      |
>  1596                  +---------------+--------------+--------------+
>  1597                  | Track Ingress | A            | A            |
>  1598                  +---------------+--------------+--------------+
>  1599                  |    TrackID    | (A, 129)     | (A, 129)     |
>  1600                  +---------------+--------------+--------------+
>  1601                  |      VIO      | C, D, E      | A, B, C      |
>  1602                  +---------------+--------------+--------------+
>  1603                  |    Targets    | E, F, G      | E, F, G      |
>  1604                  +---------------+--------------+--------------+
>  1605
>  1606                              Table 1: P-DAO Messages
> 
> Please add a line showing the SegmentID of the VIO so its clear how

If that helps..

             +===============+==============+==============+
             |     Field     | P-DAO 1 to E | P-DAO 2 to C |
             +===============+==============+==============+
             |      Mode     | Storing      | Storing      |
             +---------------+--------------+--------------+
             | Track Ingress | A            | A            |
             +---------------+--------------+--------------+
             |    TrackID    | (A, 129)     | (A, 129)     |
             +---------------+--------------+--------------+
             |   SegmentID   | 1            | 2            |
             +---------------+--------------+--------------+
             |      VIO      | C, D, E      | A, B, C      |
             +---------------+--------------+--------------+
             |    Targets    | E, F, G      | E, F, G      |
             +---------------+--------------+--------------+

> the P-DAO are distinguished. I guess that is the identifier used, right ?

Right, and it is significant for a Track ID so in NSM it can be reused as follows:

     +===============+==============+==============+==============+
     |               | P-DAO 1 to C | P-DAO 2 to A | P-DAO 3 to A |
     +===============+==============+==============+==============+
     |      Mode     | Non-Storing  | Non-Storing  | Non-Storing  |
     +---------------+--------------+--------------+--------------+
     | Track Ingress | C            | A            | A            |
     +---------------+--------------+--------------+--------------+
     |    TrackID    | (C, 131)     | (A, 129)     | (A, 141)     |
     +---------------+--------------+--------------+--------------+
     |   SegmentID   | 1            | 1            | 1            |
     +---------------+--------------+--------------+--------------+
     |      VIO      | D, E         | B, C         | E            |
     +---------------+--------------+--------------+--------------+
     |    Targets    | E            | E            | F, G         |
     +---------------+--------------+--------------+--------------+

> 
> If this example is pimped up as asked for above, it would make a good
> intro example, although it does not cover all options. But at last the
> fact that the track ingres for the secnd (C,D,E) segment is still A
> was not clear to me in before.
> 
> Also, i guess that if the track was more of a tree, then the targets
> of some earlier segments wold be a superset of the targets of later
> segments at a branching point.
> 
>  1608       As a result the RIBs are set as follows:
>  1609
>  1629
> +======+=============+=========+=============+==========+
>  1630             | Node | Destination | Origin  | Next Hop(s) | TrackID
> |
>  1631
> +======+=============+=========+=============+==========+
>  1632             |  E   | F, G        | P-DAO 1 | Neighbor    | (A,
> 129) |
> 
> But E would know about F and G alrady before P-DAO 1, so i wonder...
> 
> he example didn't call out which DODAG we're forwarding across. I guess
> this
> is a sequental Track, which the example also does not say.

It is illustrated at the beginning

Before diving deeper into Track Legs and Segments signaling and
  operation, this section provides examples of what how route
  projection works through variations of a simple example.  Say we want
  to build a Serial Track from node A to E in Figure 1, so A can route
  packets to either E or its neighbors F and G along A, B, C, D and E
  as opposed to via the Root:

                                                /===> F
                  A ===> B ===> C ===> D===> E <
                                                \===> G

                        Figure 1: Reference Track

would be good to
> say Destination in which context in the table. "Destination in RPLInstance
> TrackID1 ???"

That's a but heavy

> 
> Q: If this is not in the main DODAG, but in that TrackID1 DODAG,
> and we didn't send any P-DAO yet. Can we actually send/deliver packets ?

Without the Track the packets go along the main DODAG, IOW in NSM via the Root that adds an SRH

> I guess no, right ? So what seems to happen is that we first have some
> F, G neighbor information from the main DODAG, and because of P-DAO 1,
> this information is inherited into the forwarding for TrackID1 DODAG ?
> That would be good to show, or explain.

We could write a book, but in an RFC we end up scrubbing that sort of text.
A Segment installed from egress and connects to the target even as it is being built.
A Track Leg cannot be set up before all the segments it uses. So we're good.

> 
> Also, given how you previously whee pointing out that these routes
> are used over routes to the node because of prefix-length, should the
> entries in the destination not rather read F/128, G/128, likewise for all
> the
> other entries below ? Or does RPL imply fixed /128 unless its a /0 ?
> default
> route to the root ?

Agreed I only give /128 routes in the examples. RPL does not imply that though.

Adding 

  In this simple example we show host routes though RPL Targets can be prefixes.  

> 
> Also: TrackID is 129 i think. (A, 129) seems to be the DODAG (Identifier?).
> So maybe rename or rewrite accordingly.

OK

          +====================+==============+==============+
          |       Field        | P-DAO 1 to E | P-DAO 2 to C |
          +====================+==============+==============+
          |        Mode        | Storing      | Storing      |
          +--------------------+--------------+--------------+
          |   Track Ingress    | A            | A            |
          +--------------------+--------------+--------------+
          | (DODAGID, TrackID) | (A, 129)     | (A, 129)     |
          +--------------------+--------------+--------------+
          |     SegmentID      | 1            | 2            |
          +--------------------+--------------+--------------+
          |        VIO         | C, D, E      | A, B, C      |
          +--------------------+--------------+--------------+
          |      Targets       | E, F, G      | E, F, G      |
          +--------------------+--------------+--------------+

> 
>  1633             +------+-------------+---------+-------------+--------
> --+
>  1634             |  D   | E           | P-DAO 1 | Neighbor    | (A,
> 129) |
>  1635             +------+-------------+---------+-------------+--------
> --+
>  1636             |  "   | F, G        | P-DAO 1 | E           | (A,
> 129) |
>  1637             +------+-------------+---------+-------------+--------
> --+
>  1638             |  C   | D           | P-DAO 1 | Neighbor    | (A,
> 129) |
>  1639             +------+-------------+---------+-------------+--------
> --+
>  1640             |  "   | E, F, G     | P-DAO 1 | D           | (A,
> 129) |
>  1641             +------+-------------+---------+-------------+--------
> --+
>  1642             |  B   | C           | P-DAO 2 | Neighbor    | (A,
> 129) |
>  1643             +------+-------------+---------+-------------+--------
> --+
>  1644             |  "   | E, F, G     | P-DAO 2 | C           | (A,
> 129) |
>  1645             +------+-------------+---------+-------------+--------
> --+
>  1646             |  A   | B           | P-DAO 2 | Neighbor    | (A,
> 129) |
>  1647             +------+-------------+---------+-------------+--------
> --+
>  1648             |  A   | E, F, G     | P-DAO 2 | B           | (A,
> 129) |
>  1649             +------+-------------+---------+-------------+--------
> --+
>  1650
>  1651                                Table 2: RIB setting
>  1652
>  1653       E recognizes that it is the Track Egress because it is both
> a Target
>  1654       and a Segment Endpoint.
> 
> ... of P-DAO 1 ?

I removed that. No need.

> 
> What would happen if E was not listed as a Target in P-DAO 2 ?

OK that can be avoided if E is not an intended Target. I actually changed to represent that case.

          +====================+==============+==============+
          |       Field        | P-DAO 1 to E | P-DAO 2 to C |
          +====================+==============+==============+
          |        Mode        | Storing      | Storing      |
          +--------------------+--------------+--------------+
          |   Track Ingress    | A            | A            |
          +--------------------+--------------+--------------+
          | (DODAGID, TrackID) | (A, 129)     | (A, 129)     |
          +--------------------+--------------+--------------+
          |     SegmentID      | 1            | 2            |
          +--------------------+--------------+--------------+
          |        VIO         | C, D, E      | A, B, C      |
          +--------------------+--------------+--------------+
          |      Targets       | F, G         | F, G         |
          +--------------------+--------------+--------------+

In fact all the next hops in the VIO are implicit targets though it may cause the node to create too much state. This is why I had that "precedence" text.
I changed the term to priority to avoid the confusion you pointed out.

Also clarified before this that

     P-DAO 3 may be omitted if P-DAO 1 and 2 signal F and G as Targets.

> Would that impact the ability to deliver packets to F, G via (A, 129) ?

Not in the case of stitched if we do not have packets to E, but it is needed in other cases when the Leg to E signals F and G. This is why I chose to illustrate the case you propose, shows more stuff.

> If so, why ?  If its possible to deliver to (F,G) without indicating E in
> target list, hen i think the statement is wrong.
> 
> And i can't see how the logic works. Lets assume C has a neighbor H,
> so now we set the targets for P-DAO 2 to E,F,G,H. How would the notion
> of including C into this target list help C to decide wheher to deliver
> packets to H directly instead of leting them maybe go through further
> segments ? Aka: It seems to me that if the logic is that if you are
> sement endpoint, and any track egres for that segments P-DAO is a neighbor
> in the main dodag, then you do forward directly to that target.

C's routing table decides. Normally connected routes to neighbors win in the RIB vs routes from IGPs.
But C does not inject a RTO for H. The main Root selects what RTOs get in the P DAO

> 
> Yes/No/Maybe ? ;-))
> 
> Also: I think there should be some "Punt" line in the FIB on E for E as
> a target. And it seems to me that that Punt line would be created
> by including the segment endpoint into the target list. Maybe one
> does not want to be able to address a segment endpoint via the Track
> DODAG...
> 
>  1656       Packets originated by A to E, F, or G, do not require an
>                             ^^^ from a source X via
> 
>  1657       encapsulation.  In any fashion, the outer headers of the
> packets that
>           ^^^^^^^^^^^^^^  ^^^^^^^^^^^^^^^ remove
> 
> how about "any additional encapsulation beside the one outer encapsulation
> required for non-storage-mode" ?
> 
> Because immediately following you do write and show that outer
> encapsulation header.

You missed the originated by A piece. Clarified as follows:

  Packets originated by A to F or G do not require an encapsulation as
  the RPI can be placed in the native header chain.  For packets that
  it routes, A must encapsulate to add the RPI that signals the
  trackID; the outer headers of the packets that are forwarded along
  the Track have the following settings:

   +========+===================+===================+================+
   | Header | IPv6 Source Addr. | IPv6 Dest.  Addr. | TrackID in RPI |
   +========+===================+===================+================+
   | Outer  |         A         |       F or G      |    (A, 129)    |
   +--------+-------------------+-------------------+----------------+
   | Inner  |       X != A      |       F or G      |      N/A       |
   +--------+-------------------+-------------------+----------------+

> 
>  1657       encapsulation.  In any fashion, the outer headers of the
> packets that
>  1658       are forwarded along the Track have the following settings:
>  1659
>  1660
> +========+===================+===================+================+
>  1661        | Header | IPv6 Source Addr. | IPv6 Dest.  Addr. | TrackID
> in RPI |
>  1662
> +========+===================+===================+================+
>  1663        | Outer  |         A         |     E, F or G     |    (A,
> 129)    |
>  1664        +--------+-------------------+-------------------+---------
> -------+
>  1665        | Inner  |       X != A      |     E, F or G     |      N/A
> |
>  1666        +--------+-------------------+-------------------+---------
> -------+
> 
> Maybe add note: The Inner header is only required for X != A.

Which is what the text I added above says, we re in sync.

> 
>  1667
>  1668                          Table 3: Packet header settings
>  1669
>  1670       As an example, say that A has a packet for F.  Using the RIB
> above:
>  1671
>  1672       *  From P-DAO 2: A forwards to B and B forwards to C.
>  1673
>  1674       *  From P-DAO 1: C forwards to D and D forwards to E.
>  1675
>  1676       *  From Neighbor Cache Entry: C delivers the packet to F.
>                                         ^ E ?
> 
> But see above, your FIB shows that E -> F as part of (A, 129), whereas the
> neighbor
> cache entry is probably independent of 129, right ? So one of those
> statements
> (table or neighbor-cache-entry) is wrong.

That was not the meaning. The meaning was that it is signaled as target of that track.

> 
>  1685    9.1.2.  External routes
>                ^^^^^^^^^^^^^^^
> 
> New term not used in before. Explain.

Added

  In this example, we consider F and G as destinations that are
  external to the Track as a DODAG, as discussed in section 4.1.1. of
  [RFC9008].  We then apply the directives for encapsulating in that
  case, more in Section 6.6.

> 
>  1687       Storing-Mode P-DAO 1 and 2, and Non-Storing-Mode P-DAO 3,
> are sent to
>  1688       E, C and A, respectively, as follows:
>  1689
>  1690
> +===============+==============+==============+==============+
>  1691          |               | P-DAO 1 to E | P-DAO 2 to C | P-DAO 3
> to A |
>  1692
> +===============+==============+==============+==============+
>  1693          |      Mode     | Storing      | Storing      | Non-
> Storing  |
>  1694          +---------------+--------------+--------------+----------
> ----+
>  1695          | Track Ingress | A            | A            | A
> |
>  1696          +---------------+--------------+--------------+----------
> ----+
>  1697          |    TrackID    | (A, 129)     | (A, 129)     | (A, 129)
> |
>  1698          +---------------+--------------+--------------+----------
> ----+
>  1699          |      VIO      | C, D, E      | A, B, C      | E
> |
>  1700          +---------------+--------------+--------------+----------
> ----+
>  1701          |    Targets    | E            | E            | F, G
> |
>  1702          +---------------+--------------+--------------+----------
> ----+
> 
> Some explanation about the purpose of this example would be useful.
> 
> Seems to be something like. In this example, we want to avoid creating
> state for F,G on B,C,D. We do this by using the two storing mode segmnts
> P-DAO1 and P-DAO 2 to reach E and a non-storing segment P-DAO 3 to reach F
> and G.
> 

  In this formulation, we set up the Track Leg explicitly, which creates less
  routing state in intermediate hops at the expense of larger packets to
  accommodate source routing

> I am not clear why P-DAO 3 needs to be non-storing though. What would
> happen

Because the P DAO is passed hop by hop to the predecessor. Also it could create loops if done otherwise.
So we simplified and leave it at strict

> if the only change to above was to indicate P-DAO 3 as storing ? aka: why
> would B,C,D need to bother about any Target list of segments that they are
> not on ?
> 
>  1704                             Table 4: P-DAO Messages
>  1705
>  1706       As a result the RIBs are set as follows:
>  1707
>  1708
> +======+=============+=========+=============+==========+
>  1709             | Node | Destination | Origin  | Next Hop(s) | TrackID
> |
>  1710
> +======+=============+=========+=============+==========+
>  1711             |  E   | F, G        | P-DAO 1 | Neighbor    | (A,
> 129) |
>                                        ^^^^^^^
> 
> Shouldn't this be P-DAO 3 ??
> 
>  1712             +------+-------------+---------+-------------+--------
> --+
>  1713             |  D   | E           | P-DAO 1 | Neighbor    | (A,
> 129) |
>  1714             +------+-------------+---------+-------------+--------
> --+
>  1715             |  C   | D           | P-DAO 1 | Neighbor    | (A,
> 129) |
>  1716             +------+-------------+---------+-------------+--------
> --+
>  1717             |  "   | E           | P-DAO 1 | D           | (A,
> 129) |
>  1718             +------+-------------+---------+-------------+--------
> --+
>  1719             |  B   | C           | P-DAO 2 | Neighbor    | (A,
> 129) |
>  1720             +------+-------------+---------+-------------+--------
> --+
>  1721             |  "   | E           | P-DAO 2 | C           | (A,
> 129) |
>  1722             +------+-------------+---------+-------------+--------
> --+
>  1723             |  A   | B           | P-DAO 2 | Neighbor    | (A,
> 129) |
>  1724             +------+-------------+---------+-------------+--------
> --+
>  1725             |  A   | E           | P-DAO 2 | B           | (A,
> 129) |
>  1726             +------+-------------+---------+-------------+--------
> --+
>  1727             |  A   | F, G        | P-DAO 3 | E           | (A,
> 129) |
>  1728             +------+-------------+---------+-------------+--------
> --+
>  1729
>  1730                                Table 5: RIB setting
>  1731
>  1741       Packets from A to E do not require an encapsulation.  In any
> fashion,
>                                                 ^^^^^^^^^^^^^
> ^^^^^^^^^^^^^^  delete
>                                                 Inner Header
> 
>  1742       the outer headers of the packets that are forwarded along
> the Track
>  1743       have the following settings:
>  1744
>  1745
> +========+===================+====================+================+
>  1746       | Header | IPv6 Source Addr. | IPv6 Dest.  Addr.  | TrackID
> in RPI |
>  1747
> +========+===================+====================+================+
>  1748       | Outer  |         A         |         E          |    (A,
> 129)    |
>  1749       +--------+-------------------+--------------------+---------
> -------+
>  1750       | Inner  |         X         | E (X != A), F or G |      N/A
> |
>  1751       +--------+-------------------+--------------------+---------
> -------+
>  1752
>  1753                         Table 6: Packet header settings
>  1754
>  1755       As an example, say that A has a packet for F.  Using the RIB
> above:
>  1756
>  1757       *  From P-DAO 3: A encapsulates the packet the Track
> signaled by
>                                                     ^ for     ^ (A,129)
>  1758          P-DAO 3, with the outer header above.  Now the packet
> destination
>  1759          is E.
>             ^ of the Outer Header
>  1760
>  1761       *  From P-DAO 2: A forwards to B and B forwards to C.
>  1762
>  1763       *  From P-DAO 1: C forwards to D and D forwards to E; E
> decapsulates
>  1764          the packet.
>                        ^ because it is the destination of the Outer Header
>         ?? AND ? a valid target  for (A, 129) ??? (see question above
>  1765
>  1766       *  From Neighbor Cache Entry: C delivers packets to F or G.
>  1767
>  1768    9.1.3.  Segment Routing
> 
> New term not used in before in this doc, define, reference (RFC8402 ?),
> etc. pp.

That was the intension. If you use P-DAO in ACP, you'll probably use the RFC 8754 as opposed to RFC 8138 for the SRH, won't you?

> 
>  1770       Storing-Mode P-DAO 1 and 2, and Non-Storing-Mode P-DAO 3,
> are sent to
>  1771       E, B and A, respectively, as follows:
> 
> Again: Please add purpose/goal of this example.

Yes, added to all. In this case that gives 

  *  P-DAO 1 signals C==>D==>E-to-E

  *  P-DAO 2 signals A==>B-to-B,C

  *  P-DAO 3 signals F and G via the A-->C-->E-to-F,G Track

> 
> If you did improve initial picture to show segments, then of course these
> ar different now.
> 
> 
>  1773
> +===============+==============+==============+==============+
>  1774          |               | P-DAO 1 to E | P-DAO 2 to B | P-DAO 3
> to A |
>  1775
> +===============+==============+==============+==============+
>  1776          |      Mode     | Storing      | Storing      | Non-
> Storing  |
>  1777          +---------------+--------------+--------------+----------
> ----+
>  1778          | Track Ingress | A            | A            | A
> |
>  1779          +---------------+--------------+--------------+----------
> ----+
>  1780          |    TrackID    | (A, 129)     | (A, 129)     | (A, 129)
> |
>  1781          +---------------+--------------+--------------+----------
> ----+
>  1782          |      VIO      | C, D, E      | A, B         | C, E
> |
>  1783          +---------------+--------------+--------------+----------
> ----+
>  1784          |    Targets    | E            | B, C         | F, G
> |
>  1785          +---------------+--------------+--------------+----------
> ----+
>  1786
>  1787                             Table 7: P-DAO Messages
> 
> Does this example only work with B being the exit node for P-DAO2, or could
> it equally work if we kept P-DAO 2 unchanged, ending at C, but the target
> only being C ?

Both work, I ack the depth of your review. You really went into it. Many thanks.

I added

  Note in the above that the Segment can terminate at the loose hop as
  used in the example of P-DAO 1 or at the previous hop as done with
  P-DAO 2.  Both methods are possible on any Segment joined by a loose
  Track Leg. P-DAO 1 generates more signaling since E is the Segment
  Egress when D could be, but has the benefit that it validates that
  the connectivity between D and E still exists.

> I am guessing it could be, given how C would also need to look into P-DAO1
> route
> towards E, right ? Aka: Minimal change over prior example might make it
> easier..

I like to illustrate both cases as opposed to mandate one

> 
>  1797       As a result the RIBs are set as follows:
>  1798
>  1799
> +======+=============+=========+=============+==========+
>  1800             | Node | Destination | Origin  | Next Hop(s) | TrackID
> |
>  1801
> +======+=============+=========+=============+==========+
>  1802             |  E   | F, G        | P-DAO 1 | Neighbor    | (A,
> 129) |
>  1803             +------+-------------+---------+-------------+--------
> --+
>  1804             |  D   | E           | P-DAO 1 | Neighbor    | (A,
> 129) |
>  1805             +------+-------------+---------+-------------+--------
> --+
>  1806             |  C   | D           | P-DAO 1 | Neighbor    | (A,
> 129) |
>  1807             +------+-------------+---------+-------------+--------
> --+
>  1808             |  "   | E           | P-DAO 1 | D           | (A,
> 129) |
>  1809             +------+-------------+---------+-------------+--------
> --+
>  1810             |  B   | C           | P-DAO 2 | Neighbor    | (A,
> 129) |
>  1811             +------+-------------+---------+-------------+--------
> --+
>  1812             |  A   | B           | P-DAO 2 | Neighbor    | (A,
> 129) |
>  1813             +------+-------------+---------+-------------+--------
> --+
>  1814             |  "   | C           | P-DAO 2 | B           | (A,
> 129) |
>  1815             +------+-------------+---------+-------------+--------
> --+
>  1816             |  "   | E, F, G     | P-DAO 3 | C, E        | (A,
> 129) |
>  1817             +------+-------------+---------+-------------+--------
> --+
>  1818
>  1819                                Table 8: RIB setting
>  1820
>  1821       Packets from A to E do not require an encapsulation, but
> carry a SRH
> 
> a third header instead of encap ?

Then again I mean originated by A.

> 
>  1822       via C.  In any fashion, the outer headers of the packets
> that are
>  1823       forwarded along the Track have the following settings:
>  1824
>  1825
> +========+===================+====================+================+
>  1826       | Header | IPv6 Source Addr. | IPv6 Dest.  Addr.  | TrackID
> in RPI |
>                                          ^^^^^^^^^^^^^^^^^
> 
> the SRH for th Outer Header is not just an Pv6 Dest. Addr.
> 
>  1827
> +========+===================+====================+================+
>  1828       | Outer  |         A         |  C till C then E   |    (A,
> 129)    |
>  1829       +--------+-------------------+--------------------+---------
> -------+
>  1830       | Inner  |         X         | E (X != A), F or G |      N/A
> |
>  1831       +--------+-------------------+--------------------+---------
> -------+
>  1832
>  1833                         Table 9: Packet header settings
>  1834
>  1835       As an example, say that A has a packet for F.  Using the RIB
> above:
>  1836
>  1837       *  From P-DAO 3: A encapsulates the packet the Track
> signaled by
>  1838          P-DAO 3, with the outer header above.  Now the
> destination in the
>  1839          IPv6 Header is C, and a SRH signals the final destination
> is E.
>  1840
>  1841       *  From P-DAO 2: A forwards to B and B forwards to C.
>  1842
>  1843       *  From P-DAO 3: C processes the SRH and sets the
> destination in the
>  1844          IPv6 Header to E.
>  1845
>  1853       *  From P-DAO 1: C forwards to D and D forwards to E; E
> decapsulates
>  1854          the packet.
>  1855
>  1856       *  From the Neighbor Cache Entry: C delivers packets to F or
> G.
>  1857
>  1858    9.2.  Using Non-Storing-Mode joining Tracks
>  1859
>  1860       A==>B==>C and C==>D==>E are Tracks expressed as non-storing
> P-DAOs.
>  1861
>  1862    9.2.1.  Stitched Tracks
> 
> sequential Tracks ?

Yes, that would work too.

> 
>  1864       Non-Storing Mode P-DAO 1 and 2 are sent to C and A
> respectively, as
>  1865       follows:
>  1866
>  1867                  +===============+==============+==============+
>  1868                  |               | P-DAO 1 to C | P-DAO 2 to A |
>  1869                  +===============+==============+==============+
>  1870                  |      Mode     | Non-Storing  | Non-Storing  |
>  1871                  +---------------+--------------+--------------+
>  1872                  | Track Ingress | C            | A            |
>  1873                  +---------------+--------------+--------------+
>  1874                  |    TrackID    | (C, 131)     | (A, 129)     |
>  1875                  +---------------+--------------+--------------+
>  1876                  |      VIO      | D, E         | B, C         |
>  1877                  +---------------+--------------+--------------+
>  1878                  |    Targets    | F, G         | E, F, G      |
>  1879                  +---------------+--------------+--------------+
> 
> WOuld it be possible for both DAO to have the same number (129) given how
> they are disambiguated by the source address ? If so i think it would
> strenthen
> the example by doing so and highlighting that these are not the same Tracks
> anymore,
> even if they reuse the same TrackID.

It would. I used TrackID of 131 in both cases and added

  A encapsulates the packet with destination of F in the Track signaled by
  P-DAO 2. The outer header has source A, destination B, an SRH that
  indicates C as the next loose hop, and a RPI indicating a TrackId of 131
  from A's namespace, which is distinct from TrackId of 131 from C's.
> 
>  1880
>  1881                              Table 10: P-DAO Messages
>  1882
>  1883       As a result the RIBs are set as follows:
>  1884
>  1885
>  1886
>  1887
>  1888
>  1889
>  1890
>  1891
>  1892
>  1893
>  1894
>  1895
>  1896
>  1897
>  1898
>  1899
>  1900
>  1901
>  1902
>  1903
>  1904    Thubert, et al.          Expires 28 January 2022
> [Page 34]
>  1905
> 
> 
>  1906    Internet-Draft               DAO Projection
> July 2021
>  1907
>  1908
>  1909
> +======+=============+=========+=============+==========+
>  1910             | Node | Destination | Origin  | Next Hop(s) | TrackID
> |
>  1911
> +======+=============+=========+=============+==========+
>  1912             |  E   | F, G        | ND      | Neighbor    | Any
> |
>  1913             +------+-------------+---------+-------------+--------
> --+
>  1914             |  D   | E           | ND      | Neighbor    | Any
> |
>  1915             +------+-------------+---------+-------------+--------
> --+
>  1916             |  C   | D           | ND      | Neighbor    | Any
> |
>  1917             +------+-------------+---------+-------------+--------
> --+
>  1918             |  "   | E, F, G     | P-DAO 1 | D, E        | (C,
> 131) |
>  1919             +------+-------------+---------+-------------+--------
> --+
>  1920             |  B   | C           | ND      | Neighbor    | Any
> |
>  1921             +------+-------------+---------+-------------+--------
> --+
>  1922             |  A   | B           | ND      | Neighbor    | Any
> |
>  1923             +------+-------------+---------+-------------+--------
> --+
>  1924             |  "   | C, E, F, G  | P-DAO 2 | B, C        | (A,
> 129) |
>  1925             +------+-------------+---------+-------------+--------
> --+
>  1926
>  1927                               Table 11: RIB setting
>  1928
>  1929       Packets from A to E, F and G do not require an
> encapsulation, though
>  1930       it is preferred that A encapsulates and C decapsulates.
> Either way,
>  1931       they carry a SRH via B and C, and C needs to encapsulate to
> E, F, or
>  1932       G to add an SRH via D and E.  The encapsulating headers of
> packets
>  1933       that are forwarded along the Track between C and E have the
> following
>  1934       settings:
>  1935
>  1936
> +========+===================+===================+================+
>  1937        | Header | IPv6 Source Addr. | IPv6 Dest.  Addr. | TrackID
> in RPI |
>  1938
> +========+===================+===================+================+
>  1939        | Outer  |         C         |  D till D then E  |    (C,
> 131)    |
>  1940        +--------+-------------------+-------------------+---------
> -------+
>  1941        | Inner  |         X         |     E, F, or G    |      N/A
> |
>  1942        +--------+-------------------+-------------------+---------
> -------+
>  1943
>  1944                          Table 12: Packet header settings
>  1945
>  1946       As an example, say that A has a packet for F.  Using the RIB
> above:
>  1947
>  1948       *  From P-DAO 2: A encapsulates the packet with destination
> of F in
>  1949          the Track signaled by P-DAO 2.  The outer header has
> source A,
>  1950          destination B, an SRH that indicates C as the next loose
> hop, and
>  1951          a RPI indicating a TrackId of 129 from A's namespace.
>  1952
>  1953       *  From the SRH: Packets forwarded by B have source A,
> destination C
>  1954          , a consumed SRH, and a RPI indicating a TrackId of 129
> from A's
>  1955          namespace.  C decapsulates.
>  1956
>  1965       *  From P-DAO 1: C encapsulates the packet with destination
> of F in
>  1966          the Track signaled by P-DAO 1.  The outer header has
> source C,
>  1967          destination D, an SRH that indicates E as the next loose
> hop, and
>  1968          a RPI indicating a TrackId of 131 from C's namespace.  E
>  1969          decapsulates.
>  1970
>  1971    9.2.2.  External routes
> 
> sequential tracks with external routes ?
> 
>  1972
>  1973       Non-Storing Mode P-DAO 1 is sent to C and Non-Storing Mode
> P-DAO 2
>  1974       and 3 are sent A, as follows:
>  1975
>  1976
> +===============+==============+==============+==============+
>  1977          |               | P-DAO 1 to C | P-DAO 2 to A | P-DAO 3
> to A |
>  1978
> +===============+==============+==============+==============+
>  1979          |      Mode     | Non-Storing  | Non-Storing  | Non-
> Storing  |
>  1980          +---------------+--------------+--------------+----------
> ----+
>  1981          | Track Ingress | C            | A            | A
> |
>  1982          +---------------+--------------+--------------+----------
> ----+
>  1983          |    TrackID    | (C, 131)     | (A, 129)     | (A, 141)
> |
>  1984          +---------------+--------------+--------------+----------
> ----+
>  1985          |      VIO      | D, E         | B, C         | E
> |
>  1986          +---------------+--------------+--------------+----------
> ----+
>  1987          |    Targets    | E            | E            | F, G
> |
>  1988          +---------------+--------------+--------------+----------
> ----+
>  1989
>  1990                             Table 13: P-DAO Messages
>  1991
>  1992       As a result the RIBs are set as follows:
>  1993
>  1994
>  1995
>  1996
>  1997
>  1998
>  1999
>  2000
>  2001
>  2002
>  2003
>  2004
>  2005
>  2006
>  2007
>  2008
>  2009
>  2010
>  2011
>  2012
>  2013
>  2014
>  2015
>  2016    Thubert, et al.          Expires 28 January 2022
> [Page 36]
>  2017
> 
> 
>  2018    Internet-Draft               DAO Projection
> July 2021
>  2019
>  2020
>  2021
> +======+=============+=========+=============+==========+
>  2022             | Node | Destination | Origin  | Next Hop(s) | TrackID
> |
>  2023
> +======+=============+=========+=============+==========+
>  2024             |  E   | F, G        | ND      | Neighbor    | Any
> |
>  2025             +------+-------------+---------+-------------+--------
> --+
>  2026             |  D   | E           | ND      | Neighbor    | Any
> |
>  2027             +------+-------------+---------+-------------+--------
> --+
>  2028             |  C   | D           | ND      | Neighbor    | Any
> |
>  2029             +------+-------------+---------+-------------+--------
> --+
>  2030             |  "   | E           | P-DAO 1 | D, E        | (C,
> 131) |
>  2031             +------+-------------+---------+-------------+--------
> --+
>  2032             |  B   | C           | ND      | Neighbor    | Any
> |
>  2033             +------+-------------+---------+-------------+--------
> --+
>  2034             |  A   | B           | ND      | Neighbor    | Any
> |
>  2035             +------+-------------+---------+-------------+--------
> --+
>  2036             |  "   | C, E        | P-DAO 2 | B, C        | (A,
> 129) |
>  2037             +------+-------------+---------+-------------+--------
> --+
>  2038             |  "   | F, G        | P-DAO 3 | E           | (A,
> 141) |
>  2039             +------+-------------+---------+-------------+--------
> --+
>  2040
>  2041                               Table 14: RIB setting
>  2042
>  2043       The encapsulating headers of packets that are forwarded
> along the
>  2044       Track between C and E have the following settings:
>  2045
>  2046
> +========+===================+===================+================+
>  2047        | Header | IPv6 Source Addr. | IPv6 Dest.  Addr. | TrackID
> in RPI |
>  2048
> +========+===================+===================+================+
>  2049        | Outer  |         C         |  D till D then E  |    (C,
> 131)    |
>  2050        +--------+-------------------+-------------------+---------
> -------+
>  2051        | Middle |         A         |         E         |    (A,
> 141)    |
>  2052        +--------+-------------------+-------------------+---------
> -------+
>  2053        | Inner  |         X         |     E, F or G     |      N/A
> |
>  2054        +--------+-------------------+-------------------+---------
> -------+
>  2055
>  2056                          Table 15: Packet header settings
>  2057
>  2058       As an example, say that A has a packet for F.  Using the RIB
> above:
>  2059
>  2060       *  From P-DAO 3: A encapsulates the packet with destination
> of F in
>  2061          the Track signaled by P-DAO 3.  The outer header has
> source A,
>  2062          destination E, and a RPI indicating a TrackId of 141 from
> A's
>  2063          namespace.  This recurses with:
>  2064
>  2065       *  From P-DAO 2: A encapsulates the packet with destination
> of E in
>  2066          the Track signaled by P-DAO 2.  The outer header has
> source A,
>  2067          destination B, an SRH that indicates C as the next loose
> hop, and
>  2068          a RPI indicating a TrackId of 129 from A's namespace.
>  2069
>  2070
>  2071
>  2072    Thubert, et al.          Expires 28 January 2022
> [Page 37]
>  2073
> 
> 
>  2074    Internet-Draft               DAO Projection
> July 2021
>  2075
>  2076
>  2077       *  From the SRH: Packets forwarded by B have source A,
> destination C
>  2078          , a consumed SRH, and a RPI indicating a TrackId of 129
> from A's
>  2079          namespace.  C decapsulates.
>  2080
>  2081       *  From P-DAO 1: C encapsulates the packet with destination
> of E in
>  2082          the Track signaled by P-DAO 1.  The outer header has
> source C,
>  2083          destination D, an SRH that indicates E as the next loose
> hop, and
>  2084          a RPI indicating a TrackId of 131 from C's namespace.  E
>  2085          decapsulates.
>  2086
>  2087    9.2.3.  Segment Routing
> 
> segment routing with exernal routes ?
>  2088
>  2089       Non-Storing Mode P-DAO 1 is sent to C and Non-Storing Mode
> P-DAO 2
>  2090       and 3 are sent A, as follows:
>  2091
>  2092
> +===============+==============+==============+==============+
>  2093          |               | P-DAO 1 to C | P-DAO 2 to A | P-DAO 3
> to A |
>  2094
> +===============+==============+==============+==============+
>  2095          |      Mode     | Non-Storing  | Non-Storing  | Non-
> Storing  |
>  2096          +---------------+--------------+--------------+----------
> ----+
>  2097          | Track Ingress | C            | A            | A
> |
>  2098          +---------------+--------------+--------------+----------
> ----+
>  2099          |    TrackID    | (C, 131)     | (A, 129)     | (A, 141)
> |
>  2100          +---------------+--------------+--------------+----------
> ----+
>  2101          |      VIO      | D, E         | B            | C, E
> |
>  2102          +---------------+--------------+--------------+----------
> ----+
>  2103          |    Targets    |              | C            | F, G
> |
>  2104          +---------------+--------------+--------------+----------
> ----+
>  2105
>  2106                             Table 16: P-DAO Messages
>  2107
>  2108       As a result the RIBs are set as follows:
>  2109
>  2110
>  2111
>  2112
>  2113
>  2114
>  2115
>  2116
>  2117
>  2118
>  2119
>  2120
>  2121
>  2122
>  2123
>  2124
>  2125
>  2126
>  2127
>  2128    Thubert, et al.          Expires 28 January 2022
> [Page 38]
>  2129
> 
> 
>  2130    Internet-Draft               DAO Projection
> July 2021
>  2131
>  2132
>  2133
> +======+=============+=========+=============+==========+
>  2134             | Node | Destination | Origin  | Next Hop(s) | TrackID
> |
>  2135
> +======+=============+=========+=============+==========+
>  2136             |  E   | F, G        | ND      | Neighbor    | Any
> |
>  2137             +------+-------------+---------+-------------+--------
> --+
>  2138             |  D   | E           | ND      | Neighbor    | Any
> |
>  2139             +------+-------------+---------+-------------+--------
> --+
>  2140             |  C   | D           | ND      | Neighbor    | Any
> |
>  2141             +------+-------------+---------+-------------+--------
> --+
>  2142             |  "   | E           | P-DAO 1 | D, E        | (C,
> 131) |
>  2143             +------+-------------+---------+-------------+--------
> --+
>  2144             |  B   | C           | ND      | Neighbor    | Any
> |
>  2145             +------+-------------+---------+-------------+--------
> --+
>  2146             |  A   | B           | ND      | Neighbor    | Any
> |
>  2147             +------+-------------+---------+-------------+--------
> --+
>  2148             |  "   | C           | P-DAO 2 | B, C        | (A,
> 129) |
>  2149             +------+-------------+---------+-------------+--------
> --+
>  2150             |  "   | E, F, G     | P-DAO 3 | C, E        | (A,
> 141) |
>  2151             +------+-------------+---------+-------------+--------
> --+
>  2152
>  2153                               Table 17: RIB setting
>  2154
>  2155       The encapsulating headers of packets that are forwarded
> along the
>  2156       Track between A and B have the following settings:
>  2157
>  2158
> +========+===================+===================+================+
>  2159        | Header | IPv6 Source Addr. | IPv6 Dest.  Addr. | TrackID
> in RPI |
>  2160
> +========+===================+===================+================+
>  2161        | Outer  |         A         |  B till D then E  |    (A,
> 129)    |
>  2162        +--------+-------------------+-------------------+---------
> -------+
>  2163        | Middle |         A         |         C         |    (A,
> 141)    |
>  2164        +--------+-------------------+-------------------+---------
> -------+
>  2165        | Inner  |         X         |     E, F or G     |      N/A
> |
>  2166        +--------+-------------------+-------------------+---------
> -------+
>  2167
>  2168                          Table 18: Packet header settings
>  2169
>  2170       The encapsulating headers of packets that are forwarded
> along the
>  2171       Track between B and C have the following settings:
>  2172
>  2173
>  2174
>  2175
>  2176
>  2177
>  2178
>  2179
>  2180
>  2181
>  2182
>  2183
>  2184    Thubert, et al.          Expires 28 January 2022
> [Page 39]
>  2185
> 
> 
>  2186    Internet-Draft               DAO Projection
> July 2021
>  2187
>  2188
>  2189
> +========+===================+===================+================+
>  2190        | Header | IPv6 Source Addr. | IPv6 Dest.  Addr. | TrackID
> in RPI |
>  2191
> +========+===================+===================+================+
>  2192        | Outer  |         A         |         C         |    (A,
> 141)    |
>  2193        +--------+-------------------+-------------------+---------
> -------+
>  2194        | Inner  |         X         |     E, F or G     |      N/A
> |
>  2195        +--------+-------------------+-------------------+---------
> -------+
>  2196
>  2197                          Table 19: Packet header settings
>  2198
>  2199       The encapsulating headers of packets that are forwarded
> along the
>  2200       Track between C and E have the following settings:
>  2201
>  2202
> +========+===================+===================+================+
>  2203        | Header | IPv6 Source Addr. | IPv6 Dest.  Addr. | TrackID
> in RPI |
>  2204
> +========+===================+===================+================+
>  2205        | Outer  |         C         |  D till D then E  |    (C,
> 131)    |
>  2206        +--------+-------------------+-------------------+---------
> -------+
>  2207        | Middle |         A         |         E         |    (A,
> 141)    |
>  2208        +--------+-------------------+-------------------+---------
> -------+
>  2209        | Inner  |         X         |     E, F or G     |      N/A
> |
>  2210        +--------+-------------------+-------------------+---------
> -------+
>  2211
>  2212                          Table 20: Packet header settings
>  2213
>  2214       As an example, say that A has a packet for F.  Using the RIB
> above:
>  2215
>  2216       *  From P-DAO 3: A encapsulates the packet with destination
> of F in
>  2217          the Track signaled by P-DAO 3.  The outer header has
> source A,
>  2218          destination C, an SRH that indicates E as the next loose
> hop, and
>  2219          a RPI indicating a TrackId of 141 from A's namespace.
> This
>  2220          recurses with:
>  2221
>  2222       *  From P-DAO 2: A encapsulates the packet with destination
> of C in
>  2223          the Track signaled by P-DAO 2.  The outer header has
> source A,
>  2224          destination B, and a RPI indicating a TrackId of 129 from
> A's
>  2225          namespace.  B decapsulates forwards to C based on a
> sibling
>  2226          connected route.
>  2227
>  2228       *  From the SRH: C consumes the SRH and makes the
> destination E.
>  2229
>  2230       *  From P-DAO 1: C encapsulates the packet with destination
> of E in
>  2231          the Track signaled by P-DAO 1.  The outer header has
> source C,
>  2232          destination D, an SRH that indicates E as the next loose
> hop, and
>  2233          a RPI indicating a TrackId of 131 from C's namespace.  E
>  2234          decapsulates.
>  2235
> 
> Sorry, i skipped mostly across 1883 to here, running out of time.
> 
>  2245    10.  Security Considerations
>  2246
>  2247       It is worth noting that with [RPL], every node in the LLN is
> RPL-
>  2248       aware and can inject any RPL-based attack in the network.
> This draft
> 
> I guss even RUL nodes, which i think came after rfc6550 could be nasty too
> and
> inject RPL messages.

But they should be filtered, that's the whole point of forcing them to speak RFC 8505 only.

> 
>  2249       uses messages that are already present in RPL [RPL] with
> optional
>  2250       secured versions.  The same secured versions may be used
> with this
>  2251       draft, and whatever security is deployed for a given network
> also
>  2252       applies to the flows in this draft.
>  2253
>  2254       The LLN nodes depend on the 6LBR and the RPL participants
> for their
>  2255       operation.  A trust model must be put in place to ensure
> that the
>  2256       right devices are acting in these roles, so as to avoid
> threats such
>  2257       as black-holing, (see [RFC7416] section 7).  This trust
> model could
>  2258       be at a minimum based on a Layer-2 Secure joining and the
> Link-Layer
>  2259       security.  This is a generic 6LoWPAN requirement, see Req5.1
> in
>  2260       Appendix B.5 of [RFC8505].
> 
>  2261
>  2262       In a general manner, the Security Considerations in [RPL],
> and
>  2263       [RFC7416] apply to this specification as well.  The Link-
> Layer
>  2264       security is needed in particular to prevent Denial-Of-
> Service attacks
>  2265       whereby a rogue router creates a high churn in the RPL
> network by
>  2266       constantly injected forged P-DAO messages and using up all
> the
>  2267       available storage in the attacked routers.
> 
> Seems like the answer is no.
> 
>  2268
>  2269       Additionally, the trust model could include a role
> validation (e.g.,
>  2270       using a role-based authorization) to ensure that the node
> that claims
>  2271       to be a RPL Root is entitled to do so.  That trust should
> propagate
>  2272       from egress to ingress in the case of a Storing Mode P-DAO.
> 
> In ANIMA i am pondering to get role-based certificates to enable something
> like this.
> Is there any existing solution for this ? 

Not that I know of, let's see what Adrian's interim leads to

> If not, then it would be good to mention this
> as a gap. In general, the whole PCE based routing model does put more
> reliance of a
> trustworthy PCE into the solution, but it eliminates  a lot of security
> attacks
> coming from the normal peer-2-peer routing model (i am just making this
> argument
> in my bier-te draft, ask me again, when i have read BenK's feedback ;-).

😊

> 
> But when
> like it seems in this solution, one mixes the tradtional non-role based
> signaling
> with PCE input then it looks like a bigge gap than in before, because now
> these
> new routes allow to potentially create even more black holes than may have
> been
> possible in before (not quite sure).
> 
> In any case, you can also mention that you did try to eliminate one new
> type of
> attack by injection of duplicate addresses in tracks by mandating to
> discover such
> issue. Not sure though if this is the best set of heuristics that nodes
> could
> apply to self-defend themslfes and further hops from incorrect new
> messages. Miht
> be worth to ponder.

Added

   This specification suggests some validation of the VIO to prevent basic
   loops by avoiding that a node appears twice. But that is only a minimal
   protection. Arguably, an attacker tha can inject P-DAOs can reroute any
   traffic and deplete critical resources such as spectrum and battery in
   the LLN rapidly.

> 
>  2273
>  2274    11.  IANA Considerations
> 
> I am not going to read this (out of time). I would suggest as mentione din
> before to replace
> all numbers with TBDi and then strongly consider early IANA allocation.
> That would
> also help a lot to resolve any possible issues with any of the registration
> asks by
> mean of the IANA/expert review ensueing.

Added (Suggested)

> 
>  2276    11.1.  New Elective 6LoWPAN Routing Header Type
>  2277
>  2278       This document updates the IANA registry titled "Elective
> 6LoWPAN
>  2279       Routing Header Type" that was created for [RFC8138] and
> assigns the
>  2280       following value:
>  2281
>  2282                      +=======+=============+===============+
>  2283                      | Value | Description | Reference     |
>  2284                      +=======+=============+===============+
>  2285                      |   7   | P-RPI-6LoRH | This document |
>  2286                      +-------+-------------+---------------+
>  2287
>  2288                           Table 21: New Elective 6LoWPAN
>  2289                                Routing Header Type
>  2290
>  2291
>  2292
>  2293
>  2294
>  2295
>  2296    Thubert, et al.          Expires 28 January 2022
> [Page 41]
>  2297
> 
> 
>  2298    Internet-Draft               DAO Projection
> July 2021
>  2299
>  2300
>  2301    11.2.  New Critical 6LoWPAN Routing Header Type
>  2302
>  2303       This document updates the IANA registry titled "Critical
> 6LoWPAN
>  2304       Routing Header Type" that was created for [RFC8138] and
> assigns the
>  2305       following value:
>  2306
>  2307                      +=======+=============+===============+
>  2308                      | Value | Description | Reference     |
>  2309                      +=======+=============+===============+
>  2310                      |   7   | P-RPI-6LoRH | This document |
>  2311                      +-------+-------------+---------------+
>  2312
>  2313                           Table 22: New Critical 6LoWPAN
>  2314                                Routing Header Type
>  2315
>  2316    11.3.  New Subregistry For The RPL Option Flags
>  2317
>  2318       IANA is required to create a subregistry for the 8-bit RPL
> Option
>  2319       Flags field, as detailed in Figure 2, under the "Routing
> Protocol for
>  2320       Low Power and Lossy Networks (RPL)" registry.  The bits are
> indexed
>  2321       from 0 (leftmost) to 7.  Each bit is tracked with the
> following
>  2322       qualities:
>  2323
>  2324       *  Bit number (counting from bit 0 as the most significant
> bit)
>  2325
>  2326       *  Indication When Set
>  2327
>  2328       *  Reference
>  2329
>  2330       Registration procedure is "Standards Action" [RFC8126].  The
> initial
>  2331       allocation is as indicated in Table 26:
>  2332
>  2333
> +============+======================+===============+
>  2334               | Bit number | Indication When Set  | Reference
> |
>  2335
> +============+======================+===============+
>  2336               |     0      | Down 'O'             | [RFC6553]
> |
>  2337               +------------+----------------------+---------------
> +
>  2338               |     1      | Rank-Error (R)       | [RFC6553]
> |
>  2339               +------------+----------------------+---------------
> +
>  2340               |     2      | Forwarding-Error (F) | [RFC6553]
> |
>  2341               +------------+----------------------+---------------
> +
>  2342               |     3      | Projected-Route (P)  | This document
> |
>  2343               +------------+----------------------+---------------
> +
>  2344
>  2345                            Table 23: Initial PDR Flags
>  2346
>  2347
>  2348
>  2349
>  2350
>  2351
>  2352    Thubert, et al.          Expires 28 January 2022
> [Page 42]
>  2353
> 
> 
>  2354    Internet-Draft               DAO Projection
> July 2021
>  2355
>  2356
>  2357    11.4.  New RPL Control Codes
>  2358
>  2359       This document extends the IANA Subregistry created by RFC
> 6550 for
>  2360       RPL Control Codes as indicated in Table 24:
>  2361
>  2362
> +======+=============================+===============+
>  2363              | Code | Description                 | Reference
> |
>  2364
> +======+=============================+===============+
>  2365              | 0x09 | Projected DAO Request (PDR) | This document
> |
>  2366              +------+-----------------------------+---------------
> +
>  2367              | 0x0A | PDR-ACK                     | This document
> |
>  2368              +------+-----------------------------+---------------
> +
>  2369
>  2370                         Table 24: New RPL Control Codes
>  2371
>  2372    11.5.  New RPL Control Message Options
>  2373
>  2374       This document extends the IANA Subregistry created by RFC
> 6550 for
>  2375       RPL Control Message Options as indicated in Table 25:
>  2376
>  2377
> +=======+============================+===============+
>  2378              | Value | Meaning                    | Reference
> |
>  2379
> +=======+============================+===============+
>  2380              |  0x0B | Stateful VIO (SF-VIO)      | This document
> |
>  2381              +-------+----------------------------+---------------
> +
>  2382              |  0x0C | Source-Routed VIO (SR-VIO) | This document
> |
>  2383              +-------+----------------------------+---------------
> +
>  2384              |  0x0D | Sibling Information option | This document
> |
>  2385              +-------+----------------------------+---------------
> +
>  2386
>  2387                      Table 25: RPL Control Message Options
>  2388
>  2389    11.6.  SubRegistry for the Projected DAO Request Flags
>  2390
>  2391       IANA is required to create a registry for the 8-bit
> Projected DAO
>  2392       Request (PDR) Flags field.  Each bit is tracked with the
> following
>  2393       qualities:
>  2394
>  2395       *  Bit number (counting from bit 0 as the most significant
> bit)
>  2396
>  2397       *  Capability description
>  2398
>  2399       *  Reference
>  2400
>  2401       Registration procedure is "Standards Action" [RFC8126].  The
> initial
>  2402       allocation is as indicated in Table 26:
>  2403
>  2404
>  2405
>  2406
>  2407
>  2408    Thubert, et al.          Expires 28 January 2022
> [Page 43]
>  2409
> 
> 
>  2410    Internet-Draft               DAO Projection
> July 2021
>  2411
>  2412
>  2413
> +============+========================+===============+
>  2414              | Bit number | Capability description | Reference
> |
>  2415
> +============+========================+===============+
>  2416              |     0      | PDR-ACK request (K)    | This document
> |
>  2417              +------------+------------------------+--------------
> -+
>  2418              |     1      | Requested path should  | This document
> |
>  2419              |            | be redundant (R)       |
> |
>  2420              +------------+------------------------+--------------
> -+
>  2421
>  2422                            Table 26: Initial PDR Flags
>  2423
>  2424    11.7.  SubRegistry for the PDR-ACK Flags
>  2425
>  2426       IANA is required to create an subregistry for the 8-bit PDR-
> ACK Flags
>  2427       field.  Each bit is tracked with the following qualities:
>  2428
>  2429       *  Bit number (counting from bit 0 as the most significant
> bit)
>  2430
>  2431       *  Capability description
>  2432
>  2433       *  Reference
>  2434
>  2435       Registration procedure is "Standards Action" [RFC8126].  No
> bit is
>  2436       currently defined for the PDR-ACK Flags.
>  2437
>  2438    11.8.  Subregistry for the PDR-ACK Acceptance Status Values
>  2439
>  2440       IANA is requested to create a Subregistry for the PDR-ACK
> Acceptance
>  2441       Status values.
>  2442
>  2443       *  Possible values are 6-bit unsigned integers (0..63).
>  2444
>  2445       *  Registration procedure is "Standards Action" [RFC8126].
>  2446
>  2447       *  Initial allocation is as indicated in Table 27:
>  2448
>  2449                +-------+------------------------+---------------+
>  2450                | Value | Meaning                | Reference     |
>  2451                +-------+------------------------+---------------+
>  2452                | 0     | Unqualified acceptance | This document |
>  2453                +-------+------------------------+---------------+
>  2454
>  2455                Table 27: Acceptance values of the PDR-ACK Status
>  2456
>  2457    11.9.  Subregistry for the PDR-ACK Rejection Status Values
>  2458
>  2459       IANA is requested to create a Subregistry for the PDR-ACK
> Rejection
>  2460       Status values.
>  2461
>  2462
>  2463
>  2464    Thubert, et al.          Expires 28 January 2022
> [Page 44]
>  2465
> 
> 
>  2466    Internet-Draft               DAO Projection
> July 2021
>  2467
>  2468
>  2469       *  Possible values are 6-bit unsigned integers (0..63).
>  2470
>  2471       *  Registration procedure is "Standards Action" [RFC8126].
>  2472
>  2473       *  Initial allocation is as indicated in Table 28:
>  2474
>  2475                 +-------+-----------------------+---------------+
>  2476                 | Value | Meaning               | Reference     |
>  2477                 +-------+-----------------------+---------------+
>  2478                 | 0     | Unqualified rejection | This document |
>  2479                 +-------+-----------------------+---------------+
>  2480
>  2481                  Table 28: Rejection values of the PDR-ACK Status
>  2482
>  2483    11.10.  SubRegistry for the Via Information Options Flags
>  2484
>  2485       IANA is requested to create a Subregistry for the 5-bit Via
>  2486       Information Options (Via Information Option) Flags field.
> Each bit
>  2487       is tracked with the following qualities:
>  2488
>  2489       *  Bit number (counting from bit 0 as the most significant
> bit)
>  2490
>  2491       *  Capability description
>  2492
>  2493       *  Reference
>  2494
>  2495       Registration procedure is "Standards Action" [RFC8126].  No
> bit is
>  2496       currently defined for the Via Information Options (Via
> Information
>  2497       Option) Flags.
>  2498
>  2499    11.11.  SubRegistry for the Sibling Information Option Flags
>  2500
>  2501       IANA is required to create a registry for the 5-bit Sibling
>  2502       Information Option (SIO) Flags field.  Each bit is tracked
> with the
>  2503       following qualities:
>  2504
>  2505       *  Bit number (counting from bit 0 as the most significant
> bit)
>  2506
>  2507       *  Capability description
>  2508
>  2509       *  Reference
>  2510
>  2511       Registration procedure is "Standards Action" [RFC8126].  The
> initial
>  2512       allocation is as indicated in Table 29:
>  2513
>  2514
>  2515
>  2516
>  2517
>  2518
>  2519
>  2520    Thubert, et al.          Expires 28 January 2022
> [Page 45]
>  2521
> 
> 
>  2522    Internet-Draft               DAO Projection
> July 2021
>  2523
>  2524
>  2525
> +============+===================================+===============+
>  2526        | Bit number | Capability description            |
> Reference     |
>  2527
> +============+===================================+===============+
>  2528        |     0      | Connectivity is bidirectional (B) | This
> document |
>  2529        +------------+-----------------------------------+---------
> ------+
>  2530
>  2531                           Table 29: Initial SIO Flags
>  2532
>  2533    11.12.  New Destination Advertisement Object Flag
>  2534
>  2535       This document modifies the "Destination Advertisement Object
> (DAO)
>  2536       Flags" registry initially created in Section 20.11 of [RPL]
> .
>  2537
>  2538       Section 3.1 also defines one new entry in the Registry as
> follows:
>  2539
>  2540              +---------------+------------------------+-----------
> +
>  2541              | Bit Number    | Capability Description | Reference
> |
>  2542              +---------------+------------------------+-----------
> +
>  2543              | 2 (suggested) | Projected DAO (P)      | THIS RFC
> |
>  2544              +---------------+------------------------+-----------
> +
>  2545
>  2546                  Table 30: New Destination Advertisement Object
>  2547                                    (DAO) Flag
>  2548
>  2549    11.13.  Error in Projected Route ICMPv6 Code
>  2550
>  2551       In some cases RPL will return an ICMPv6 error message when a
> message
>  2552       cannot be forwarded along a Projected Route.  This ICMPv6
> error
>  2553       message is "Error in Projected Route".
>  2554
>  2555       IANA has defined an ICMPv6 "Code" Fields Registry for ICMPv6
> Message
>  2556       Types.  ICMPv6 Message Type 1 describes "Destination
> Unreachable"
>  2557       codes.  This specification requires that a new code is
> allocated from
>  2558       the ICMPv6 Code Fields Registry for ICMPv6 Message Type 1,
> for "Error
>  2559       in Projected Route", with a suggested code value of 8, to be
>  2560       confirmed by IANA.
>  2561
>  2562    12.  Acknowledgments
>  2563
>  2564       The authors wish to acknowledge JP Vasseur, Remy Liubing,
> James
>  2565       Pylakutty and Patrick Wetterwald for their contributions to
> the ideas
>  2566       developed here.
>  2567
>  2568    13.  Normative References
>  2569
>  2570
>  2571
>  2572
>  2573
>  2574
>  2575
>  2576    Thubert, et al.          Expires 28 January 2022
> [Page 46]
>  2577
> 
> 
>  2578    Internet-Draft               DAO Projection
> July 2021
>  2579
>  2580
>  2581       [RFC2119]  Bradner, S., "Key words for use in RFCs to
> Indicate
>  2582                  Requirement Levels", BCP 14, RFC 2119,
>  2583                  DOI 10.17487/RFC2119, March 1997,
>  2584                  <https://www.rfc-editor.org/info/rfc2119>.
>  2585
>  2586       [RFC4443]  Conta, A., Deering, S., and M. Gupta, Ed.,
> "Internet
>  2587                  Control Message Protocol (ICMPv6) for the
> Internet
>  2588                  Protocol Version 6 (IPv6) Specification", STD 89,
>  2589                  RFC 4443, DOI 10.17487/RFC4443, March 2006,
>  2590                  <https://www.rfc-editor.org/info/rfc4443>.
>  2591
>  2592       [RFC6282]  Hui, J., Ed. and P. Thubert, "Compression Format
> for IPv6
>  2593                  Datagrams over IEEE 802.15.4-Based Networks", RFC
> 6282,
>  2594                  DOI 10.17487/RFC6282, September 2011,
>  2595                  <https://www.rfc-editor.org/info/rfc6282>.
>  2596
>  2597       [RPL]      Winter, T., Ed., Thubert, P., Ed., Brandt, A.,
> Hui, J.,
>  2598                  Kelsey, R., Levis, P., Pister, K., Struik, R.,
> Vasseur,
>  2599                  JP., and R. Alexander, "RPL: IPv6 Routing
> Protocol for
>  2600                  Low-Power and Lossy Networks", RFC 6550,
>  2601                  DOI 10.17487/RFC6550, March 2012,
>  2602                  <https://www.rfc-editor.org/info/rfc6550>.
>  2603
>  2604       [RFC6553]  Hui, J. and JP. Vasseur, "The Routing Protocol
> for Low-
>  2605                  Power and Lossy Networks (RPL) Option for
> Carrying RPL
>  2606                  Information in Data-Plane Datagrams", RFC 6553,
>  2607                  DOI 10.17487/RFC6553, March 2012,
>  2608                  <https://www.rfc-editor.org/info/rfc6553>.
>  2609
>  2610       [RFC6554]  Hui, J., Vasseur, JP., Culler, D., and V. Manral,
> "An IPv6
>  2611                  Routing Header for Source Routes with the Routing
> Protocol
>  2612                  for Low-Power and Lossy Networks (RPL)", RFC
> 6554,
>  2613                  DOI 10.17487/RFC6554, March 2012,
>  2614                  <https://www.rfc-editor.org/info/rfc6554>.
>  2615
>  2616       [RFC8174]  Leiba, B., "Ambiguity of Uppercase vs Lowercase
> in RFC
>  2617                  2119 Key Words", BCP 14, RFC 8174, DOI
> 10.17487/RFC8174,
>  2618                  May 2017, <https://www.rfc-
> editor.org/info/rfc8174>.
>  2619
>  2620       [RFC8126]  Cotton, M., Leiba, B., and T. Narten, "Guidelines
> for
>  2621                  Writing an IANA Considerations Section in RFCs",
> BCP 26,
>  2622                  RFC 8126, DOI 10.17487/RFC8126, June 2017,
>  2623                  <https://www.rfc-editor.org/info/rfc8126>.
>  2624
>  2625    14.  Informative References
>  2626
>  2627
>  2628
>  2629
>  2630
>  2631
>  2632    Thubert, et al.          Expires 28 January 2022
> [Page 47]
>  2633
> 
> 
>  2634    Internet-Draft               DAO Projection
> July 2021
>  2635
>  2636
>  2637       [RFC7102]  Vasseur, JP., "Terms Used in Routing for Low-
> Power and
>  2638                  Lossy Networks", RFC 7102, DOI 10.17487/RFC7102,
> January
>  2639                  2014, <https://www.rfc-editor.org/info/rfc7102>.
>  2640
>  2641       [RFC6997]  Goyal, M., Ed., Baccelli, E., Philipp, M.,
> Brandt, A., and
>  2642                  J. Martocci, "Reactive Discovery of Point-to-
> Point Routes
>  2643                  in Low-Power and Lossy Networks", RFC 6997,
>  2644                  DOI 10.17487/RFC6997, August 2013,
>  2645                  <https://www.rfc-editor.org/info/rfc6997>.
>  2646
>  2647       [RFC7416]  Tsao, T., Alexander, R., Dohler, M., Daza, V.,
> Lozano, A.,
>  2648                  and M. Richardson, Ed., "A Security Threat
> Analysis for
>  2649                  the Routing Protocol for Low-Power and Lossy
> Networks
>  2650                  (RPLs)", RFC 7416, DOI 10.17487/RFC7416, January
> 2015,
>  2651                  <https://www.rfc-editor.org/info/rfc7416>.
>  2652
>  2653       [6TiSCH-ARCHI]
>  2654                  Thubert, P., Ed., "An Architecture for IPv6 over
> the Time-
>  2655                  Slotted Channel Hopping Mode of IEEE 802.15.4
> (6TiSCH)",
>  2656                  RFC 9030, DOI 10.17487/RFC9030, May 2021,
>  2657                  <https://www.rfc-editor.org/info/rfc9030>.
>  2658
>  2659       [RAW-ARCHI]
>  2660                  Thubert, P., Papadopoulos, G. Z., and L. Berger,
> "Reliable
>  2661                  and Available Wireless Architecture/Framework",
> Work in
>  2662                  Progress, Internet-Draft, draft-ietf-raw-
> architecture-00,
>  2663                  12 July 2021,
> <https://datatracker.ietf.org/doc/html/
>  2664                  draft-ietf-raw-architecture-00>.
>  2665
>  2666       [TURN-ON_RFC8138]
>  2667                  Thubert, P., Ed. and L. Zhao, "A Routing Protocol
> for Low-
>  2668                  Power and Lossy Networks (RPL) Destination-
> Oriented
>  2669                  Directed Acyclic Graph (DODAG) Configuration
> Option for
>  2670                  the 6LoWPAN Routing Header", RFC 9035,
>  2671                  DOI 10.17487/RFC9035, April 2021,
>  2672                  <https://www.rfc-editor.org/info/rfc9035>.
>  2673
>  2674       [RFC8655]  Finn, N., Thubert, P., Varga, B., and J. Farkas,
>  2675                  "Deterministic Networking Architecture", RFC
> 8655,
>  2676                  DOI 10.17487/RFC8655, October 2019,
>  2677                  <https://www.rfc-editor.org/info/rfc8655>.
>  2678
>  2679       [RFC8025]  Thubert, P., Ed. and R. Cragie, "IPv6 over Low-
> Power
>  2680                  Wireless Personal Area Network (6LoWPAN) Paging
> Dispatch",
>  2681                  RFC 8025, DOI 10.17487/RFC8025, November 2016,
>  2682                  <https://www.rfc-editor.org/info/rfc8025>.
>  2683
>  2684
>  2685
>  2686
>  2687
>  2688    Thubert, et al.          Expires 28 January 2022
> [Page 48]
>  2689
> 
> 
>  2690    Internet-Draft               DAO Projection
> July 2021
>  2691
>  2692
>  2693       [RFC8138]  Thubert, P., Ed., Bormann, C., Toutain, L., and
> R. Cragie,
>  2694                  "IPv6 over Low-Power Wireless Personal Area
> Network
>  2695                  (6LoWPAN) Routing Header", RFC 8138, DOI
> 10.17487/RFC8138,
>  2696                  April 2017, <https://www.rfc-
> editor.org/info/rfc8138>.
>  2697
>  2698       [RFC8505]  Thubert, P., Ed., Nordmark, E., Chakrabarti, S.,
> and C.
>  2699                  Perkins, "Registration Extensions for IPv6 over
> Low-Power
>  2700                  Wireless Personal Area Network (6LoWPAN) Neighbor
>  2701                  Discovery", RFC 8505, DOI 10.17487/RFC8505,
> November 2018,
>  2702                  <https://www.rfc-editor.org/info/rfc8505>.
>  2703
>  2704       [USEofRPLinfo]
>  2705                  Robles, M.I., Richardson, M., and P. Thubert,
> "Using RPI
>  2706                  Option Type, Routing Header for Source Routes,
> and IPv6-
>  2707                  in-IPv6 Encapsulation in the RPL Data Plane", RFC
> 9008,
>  2708                  DOI 10.17487/RFC9008, April 2021,
>  2709                  <https://www.rfc-editor.org/info/rfc9008>.
>  2710
>  2711       [PCE]      IETF, "Path Computation Element",
>  2712                  <https://datatracker.ietf.org/doc/charter-ietf-
> pce/>.
>  2713
>  2714    Appendix A.  Applications
> 
> I wouldn't call this applications, its more like optimiations goals or the
> like
>  2715
>  2716    A.1.  Loose Source Routing
> 
> Optimizing packet size vs. routing state via Loose Source Routing
> 
>  2718       A RPL implementation operating in a very constrained LLN
> typically
>  2719       uses the Non-Storing Mode of Operation as represented in
> Figure 12.
>  2720       In that mode, a RPL node indicates a parent-child
> relationship to the
>  2721       Root, using a Destination Advertisement Object (DAO) that is
> unicast
>  2722       from the node directly to the Root, and the Root typically
> builds a
>  2723       source routed path to a destination down the DODAG by
> recursively
>  2724       concatenating this information.
>  2725
>  2726                  ------+---------
>  2727                        |          Internet
>  2728                        |
>  2729                     +-----+
>  2730                     |     | Border Router
>  2731                     |     |  (RPL Root)
>  2732                     +-----+                      ^     |        |
>  2733                        |                         | DAO | ACK    |
>  2734                  o    o   o    o                 |     |        |
> Strict
>  2735              o o   o  o   o  o  o o   o          |     |        |
> Source
>  2736             o  o o  o o    o   o   o  o  o       |     |        |
> Route
>  2737             o   o    o  o     o  o    o  o  o    |     |        |
>  2738            o  o   o  o   o         o   o o       |     v        v
>  2739            o          o             o     o
>  2740                              LLN
>  2741
>  2742
>  2743
>  2744    Thubert, et al.          Expires 28 January 2022
> [Page 49]
>  2745
> 
> 
>  2746    Internet-Draft               DAO Projection
> July 2021
>  2747
>  2748
>  2749                    Figure 12: RPL Non-Storing Mode of operation
>  2750
>  2751       Based on the parent-children relationships expressed in the
> non-
>  2752       storing DAO messages,the Root possesses topological
> information about
>  2753       the whole network, though this information is limited to the
>  2754       structure of the DODAG for which it is the destination.  A
> packet
>  2755       that is generated within the domain will always reach the
> Root, which
>  2756       can then apply a source routing information to reach the
> destination
>  2757       if the destination is also in the DODAG.  Similarly, a
> packet coming
>  2758       from the outside of the domain for a destination that is
> expected to
>  2759       be in a RPL domain reaches the Root.
>  2760
>  2761       It results that the Root, or then some associated
> centralized
>  2762       computation engine such as a PCE, can determine the amount
> of packets
>  2763       that reach a destination in the RPL domain, and thus the
> amount of
>  2764       energy and bandwidth that is wasted for transmission,
> between itself
>  2765       and the destination, as well as the risk of fragmentation,
> any
>  2766       potential delays because of a paths longer than necessary
> (shorter
>  2767       paths exist that would not traverse the Root).
>  2768
>  2769       As a network gets deep, the size of the source routing
> header that
>  2770       the Root must add to all the downward packets becomes an
> issue for
>  2771       nodes that are many hops away.  In some use cases, a RPL
> network
>  2772       forms long lines and a limited amount of well-Targeted
> routing state
>  2773       would allow to make the source routing operation loose as
> opposed to
>  2774       strict, and save packet size.
> 
> Maybe ad to be more explicit:
> 
> And example of such well-targeted routing state is what is needed to reach
> the
> neared loose steerig hops.
> 
>  2774       Limiting the packet size is directly
>  2775       beneficial to the energy budget, but, mostly, it reduces the
> chances
>  2776       of frame loss and/or packet fragmentation, which is highly
>  2777       detrimental to the LLN operation.  Because the capability to
> store a
>  2778       routing state in every node is limited, the decision of
> which route
>  2779       is installed where can only be optimized with a global
> knowledge of
>  2780       the system, a knowledge that the Root or an associated PCE
> may
>  2781       possess by means that are outside of the scope of this
> specification.
>  2782
>  2783       This specification enables to store a Storing Mode state in
>  2784       intermediate routers, which enables to limit the excursion
> of the
>  2785       source route headers in deep networks.  Once a P-DAO
> exchange has
>  2786       taken place for a given Target, if the Root operates in non
> Storing
>  2787       Mode, then it may elide the sequence of routers that is
> installed in
>  2788       the network from its source route headers to destination
> that are
>  2789       reachable via that Target, and the source route headers
> effectively
>  2790       become loose.
>  2791
>  2792
>  2793
>  2794
>  2795
>  2796
>  2797
>  2798
>  2799
>  2800    Thubert, et al.          Expires 28 January 2022
> [Page 50]
>  2801
> 
> 
>  2802    Internet-Draft               DAO Projection
> July 2021
>  2803
>  2804
>  2805    A.2.  Transversal Routes
>  2806
>  2807       RPL is optimized for Point-to-Multipoint (P2MP) and
> Multipoint-to-
>  2808       Point (MP2P), whereby routes are always installed along the
> RPL DODAG
>  2809       respectively from and towards the DODAG Root.  Transversal
> Peer to
>  2810       Peer (P2P) routes in a RPL network will generally suffer
> from some
>  2811       elongated (stretched) path versus the best possible path,
> since
>  2812       routing between 2 nodes always happens via a common parent,
> as
>  2813       illustrated in Figure 13:
> 
> That figure needs to be improved if anyone who doesn't know what a
> transversal
> route is is expected to understand it. Just paint a picture with the
> minimum
> number of nodes to show a transversal route not part of the DODAG.
> Figure 14 is kinda ok., but in 13 its really unclear where traffic flows
> and why.
> 

I reworked a bit and called them East West to align to RAW

> Also: Is there any RPL messaging that would allow for the Root to have
> enough
> knowledge about transversal connectibity not part of the DODAG, which it
> could
> then tell the PCE ? If not, then for fairness, it would be good to
> highlight this as
> a gap for the solution.

We use siblings Info option

> 
>  2815       *  In Storing Mode, unless the destination is a child of the
> source,
>  2816          the packets will follow the default route up the DODAG as
> well.
>  2817          If the destination is in the same DODAG, they will
> eventually
>  2818          reach a common parent that has a route to the
> destination; at
>  2819          worse, the common parent may also be the Root.  From that
> common
>  2820          parent, the packet will follow a path down the DODAG that
> is
>  2821          optimized for the Objective Function that was used to
> build the
>  2822          DODAG.
>  2823
>  2824       *  in Non-Storing Mode, all packets routed within the DODAG
> flow all
>  2825          the way up to the Root of the DODAG.  If the destination
> is in the
>  2826          same DODAG, the Root must encapsulate the packet to place
> an RH
>  2827          that has the strict source route information down the
> DODAG to the
>  2828          destination.  This will be the case even if the
> destination is
>  2829          relatively close to the source and the Root is relatively
> far off.
>  2830
>  2831
>  2832                          ------+---------
>  2833                           |          Internet
>  2834                           |
>  2835                        +-----+
>  2836                        |     | Border Router
>  2837                        |     |  (RPL Root)
>  2838                        +-----+
>  2839                           X
>  2840                     ^    v   o    o
>  2841                 ^ o   o  v   o  o  o o   o
>  2842                ^  o o  o v    o   o   o  o  o
>  2843                ^   o    o  v     o  o    o  o  o
>  2844               S  o   o  o   D         o   o o
>  2845               o          o             o     o
>  2846                                 LLN
>  2847
>  2848           Figure 13: Routing Stretch between S and D via common
> parent X
>  2849
>  2850       It results that it is often beneficial to enable transversal
> P2P
>  2851       routes, either if the RPL route presents a stretch from
> shortest
>  2852       path, or if the new route is engineered with a different
> objective,
>  2853
>  2854
>  2855
>  2856    Thubert, et al.          Expires 28 January 2022
> [Page 51]
>  2857
> 
> 
>  2858    Internet-Draft               DAO Projection
> July 2021
>  2859
>  2860
>  2861       and that it is even more critical in Non-Storing Mode than
> it is in
>  2862       Storing Mode, because the routing stretch is wider.  For
> that reason,
>  2863       earlier work at the IETF introduced the "Reactive Discovery
> of
>  2864       Point-to-Point Routes in Low Power and Lossy Networks"
> [RFC6997],
>  2865       which specifies a distributed method for establishing
> optimized P2P
>  2866       routes.  This draft proposes an alternate based on a
> centralized
>  2867       route computation.
>  2868
>  2869                     ------+---------
>  2870                           |          Internet
>  2871                           |
>  2872                        +-----+
>  2873                        |     | Border Router
>  2874                        |     |  (RPL Root)
>  2875                        +-----+
>  2876                           |
>  2877                     o    o   o    o
>  2878                 o o   o  o   o  o  o o   o
>  2879                o  o o  o o    o   o   o  o  o
>  2880                o   o    o  o     o  o    o  o  o
>  2881               S>>A>>>B>>C>>>D         o   o o
>  2882               o          o             o     o
>  2883                                 LLN
>  2884
>  2885                       Figure 14: Projected Transversal Route
>  2886
>  2887       This specification enables to store source-routed or Storing
> Mode
>  2888       state in intermediate routers, which enables to limit the
> stretch of
>  2889       a P2P route and maintain the characteristics within a given
> SLA.  An
>  2890       example of service using this mechanism oculd be a control
> loop that
>  2891       would be installed in a network that uses classical RPL for
>  2892       asynchronous data collection.  In that case, the P2P path
> may be
>  2893       installed in a different RPL Instance, with a different
> objective
>  2894       function.
>  2895
>  2896    Authors' Addresses
>  2897
>  2898       Pascal Thubert (editor)
>  2899       Cisco Systems, Inc
>  2900       Building D
>  2901       45 Allee des Ormes - BP1200
>  2902       06254 Mougins - Sophia Antipolis
>  2903       France
>  2904
>  2905       Phone: +33 497 23 26 34
>  2906       Email: [email-masked]
>  2907
>  2908
>  2909
>  2910
>  2911
>  2912    Thubert, et al.          Expires 28 January 2022
> [Page 52]
>  2913
> 
> 
>  2914    Internet-Draft               DAO Projection
> July 2021
>  2915
>  2916
>  2917       Rahul Arvind Jadhav
>  2918       Huawei Tech
>  2919       Kundalahalli Village, Whitefield,
>  2920       Bangalore 560037
>  2921       Karnataka
>  2922       India
>  2923
>  2924       Phone: +91-080-49160700
>  2925       Email: [email-masked]
>  2926
>  2927
>  2928       Matthew Gillmore
>  2929       Itron, Inc
>  2930       Building D
>  2931       2111 N Molter Road
>  2932       Liberty Lake,  99019
>  2933       United States
>  2934
>  2935       Phone: +1.800.635.5461
>  2936       Email: [email-masked]
>  2937
>  2938
>  2939
>  2940
>  2941
>  2942
>  2943
>  2944

Whao, that was a huge one Toerless. You've been incredibly deep in your understanding and yes, you were correct all the way.

I made a great many changes an lotta reorg. The diffs will be hard to evaluate.

Again, many thanks!

Pascal