Subject: [Rift] RIFT-01 questions and comments

From: "Jeffrey (Zhaohui) Zhang" <[email-masked]>
Date: Tue, 1 May 2018 15:50:41 +0000

Hi Tony,

I have the following questions and comments:

   Unless Section 4.2.9 is used, each node is provisioned with the level
   at which it is operating and its PoD (or otherwise a default level
   and "undefined" PoD are assumed; meaning that leafs do not need to be
   configured at all if initial configuration values are all left at 0).
   Nodes in the spine are configured with "any" PoD which has the same
   value "undefined" PoD hence we will talk about "undefined/any" PoD.

Is it that w/o ZTP, the following apply:

- only the leaf nodes can be unconfigured - they will take default level 0 and PoD "undefined"
- spines and middle nodes must have level configured
- spines must have pod unconfigured
- nodes right below the spine must have pod configured; lower nodes/leaves can have PoD derived

On the contrary, w/ ZTP, leaves must be configured with level 0 (explicitly or implicitly via leaf_only or l2l flag)?

   A node tries to form a three way adjacency if and only if
   (definitions of LEAF_ONLY are found in Section 4.2.9) ...

The draft does not distinguish between neighbors and adjacency. I think it's better to do so as in OSPF case:
        Neighboring routers
            Two routers that have interfaces to a common network.
            Neighbor relationships are maintained by, and usually
            dynamically discovered by, OSPF's Hello Protocol.

        Adjacency
            A relationship formed between selected neighboring routers
            for the purpose of exchanging routing information.  Not
            every pair of neighboring routers become adjacent.

The three-way handshake establishes neighborship; after that, level/pod is derived and then based on that, we determine if adjacency should be brought up.

   A node configured with "undefined" PoD membership MUST, after
   building first northbound three way adjacencies to a node being in a
   defined PoD, advertise that PoD as part of its LIEs.  In case that
   adjacency is lost, from all available northbound three way
   adjacencies the node with the highest System ID and defined PoD is
   chosen.  That way the northmost defined PoD value (normally the top
   spines in a PoD) can diffuse southbound towards the leafs "forcing"
   the PoD value on any node with "undefined" PoD.

Don't understand the text in the last sentence above. The spines don't have PoD configured (they connect to multiple PoDs), right? If we change "the top spines" to "the top node right below a spine" then it makes sense.

   More precisely, a spine node represents two different databases to
   its neighbors depending whether it advertises TIEs to the north or to
   the south/sideways.

A spine node does not have north adjacencies. Do you mean "a middle node" in the above paragraph?

   The N-TIEs hold all of the node's adjacencies, local prefixes and
   northbound policy-guided prefixes while the S-TIEs hold only all of
   the node's adjacencies and the default prefix with necessary
   disaggregated prefixes and southbound policy-guided prefixes.  

Why don't S-TIEs contain local prefixes? Wouldn't that lead to better routes on its southern neighbors to those local prefixes?

Layer vs. level are used in mix. Should be consistent.

        Spine21 S-TIEs:
        Node S-TIE:
          NodeElement(layer=2, neighbors((Node111, layer 1, cost 1),
          (Node112, layer 1, cost 1), (Node121, layer 1, cost 1),
          (Node122, layer 1, cost 1)))

Link information (like in the case for Node 111) should be added to the above.
Should the cost info be associated with links?

Node 121: Node S-TIEs do not have link info shown. Better be consistent.

What's different between NodeElement and NodeLinkElement?

   The mechanism used to distribute TIEs is the well-known (albeit
   modified in several respects to address fat tree requirements)
   flooding mechanism used by today's link-state protocols.  Albeit
   initially more demanding to implement it avoids many problems with
   diffused computation update style used by path vector.  As described
   before, TIEs themselves are transported over UDP with the ports
   indicates in the LIE exchanges and using the destination address (for
   unnumbered IPv4 interfaces same considerations apply as in equivalent
   OSPF case) on which the LIE adjacency has been formed.

Need to spell out the details for unnumbered IPv4 interface case instead of referring to OSPF.

   Every N-TIE is flooded northbound, providing a node at a given level
   with the complete topology of the Clos or Fat Tree network underneath
   it, including all specific prefixes.  This means that a packet
   received from a node at the same or lower level whose destination is
   covered by one of those specific prefixes may be routed directly
   towards the node advertising that prefix rather than sending the
   packet to a node at a higher level.

"received from a node at the same or lower level" should be removed;
No matter where the packet is received from, the rest of the text applies.

4.2.5.1.  Northbound SPF  <--- clarify it's for traffic going north
   N-SPF uses northbound and east-west adjacencies in North Node TIEs   <-- "Node S-TIEs"?
   when progressing Dijkstra.  Observe that this is really just a one
   hop variety since South Node TIEs are not re-flooded southbound           <--- "Node S-TIEs"?
   beyond a single level (or east-west) and with that the computation
   cannot progress beyond adjacent nodes.
   ...
   For N-SPF we are using the South Node TIEs to find according   <--- "Node S-TIEs"?
   adjacencies to verify backlink connectivity.  Just as in case of IS-
   IS or OSPF, two unidirectional links are associated together to
   confirm bidirectional connectivity.

4.2.5.2.  Southbound SPF

   S-SPF uses only the southbound adjacencies in the south node TIEs,  <-- "Node N-TIEs"?
   i.e. progresses towards nodes at lower levels.  Observe that E-W
   adjacencies are NEVER used in the computation.  This enforces the
   requirement that a packet traversing in a southbound direction must
   never change its direction.

   S-SPF uses northbound adjacencies in north node TIEs to verify  <-- "Node N-TIEs"?
   backlink connectivity.

   ...
   As further consideration, the moment A02 looses link N2 the situation
   evolves again.  A01 will have no more northbound reachability while
   still seeing A03 advertising northbound adjacencies in its south node   <--- "Node S-TIE"?
   tie.  With that it will stop advertising a default route due to
   Section 4.2.3.7.  Moreover, A02 may now inject its loopback address
   as south PGP.

"north/south node TIE" is ambiguous in that it could be interpreted as "north/south node's TIE" (TIE of the north/south node), hence the suggested change to be stringent.

       Observe that policy-guided prefixes are NOT affected
       since their scope is controlled by configuration.

What does the "since ..." part mean? What's the point of that sentence? Is it that "PGP are not considered"?

   To summarize the above in simplest terms: if a node detects that its
   default route encompasses prefixes for which one of the other nodes
   in its level has no possible next-hops in the level below, it has to
   disaggregate it to prevent black-holing or suboptimal routing.

"its default route encompasses" is unnecessary? Black-holing or sub-optimal forwarding is not caused by this node's default route. Rather, it is caused by the default route of the other nodes that has no possible next-hops below. Even if this nodes does not have a default route, the problem still exist and the solution is the same.

   1.  Create partial_neighbors = (empty), a set of neighbors with
       partial connectivity to the node X's layer from X's perspective.
       Each entry is a list of south neighbor of X and a list of nodes
       of X.layer that can't reach that neighbor.

How about the following?

   1.  Create partial_neighbors = (empty), a set of south neighbors with
       partial connectivity to the node X's layer from X's perspective.
       Each entry is a south neighbor of X and a list of nodes
       of X.layer that can't reach that neighbor.

For the following:

          for each S-TIE  <--- add "originated by Y"
              if (S-TIE.layer == X.layer and
                  X shares at least one S-neighbor with X)  <-- "Y shares at least ..."
                add S-TIE.originator to nodes_same_layer  <-- "add Y to ..."
                end if
              end for

            for each next-hop-set NHS
              isolated_nodes = nodes_same_layer
              for each NH in NHS
                if NH in partial_neighbors
                  isolated_nodes = intersection(isolated_nodes,
                                                partial_neighbors[NH].nodes)

Why do we need to calculate nodes_same_layer and initialize isolated_nodes to it? The way partial_neighbors is calculated, partial_neighbors[NH].nodes is already part of the nodes_same_layer? It seems that we could simply initialize isolated_nodes to partial_neighbors[first NH in NHS].nodes?

            copy disaggregated_prefixes to X's S-TIE
            if X's S-TIE is different
              schedule S-TIE for flooding

"replace disaggregated prefixes in X's Prefix S-TIE with disaggregated_prefixes"?

   Each disaggregated prefix is sent with the accurate path_distance.
   This allows a node to send the same S-TIE to each south neighbor.
   The south neighbor which is connected to that prefix will thus have a
   shorter path.

How about rewording to the following?

  Each disaggregated prefix is sent with the accurate path_distance to
  all direct south neighbors, including the one connected to that
  prefix (directly or indirectly), which will be able to select the right path
  (not through the neighbor advertising the disaggregated prefix) based
  on the advertised accurate path_distance.

-----------------------------------------

What does the following bullet mean?

   6.  The fabric is still supporting maximum load balancing properties
       while not trying to send traffic northbound unless necessary.

The "while ..." clause is especially confusing. W/ or w/o disaggregation, we don't send traffic northbound unless necessary, right?

----------------------------------------

4.2.9.  Optional Autoconfiguration

Is it true that:

- This section is only about level, not about PoD
- This section is only about middle nodes

I really don't get the following:

    First, we need to anchor the "top" of the cabling and that's what the
   SUPERSPINE_FLAG at node A is for.  Then things look smooth until we
   have to decide whether node Y is at the same level as I, J or at the
   same level as Y and consequently, X is south of it.

What does it mean by "whether node Y is ... at the same level as Y". Is the following what you mean?

   Whether node Y is at the same level as I, J and consequently X is south of it,
   Or whether node Y is at the same level as F.

For the following:

   5.  A node that changed its defined level value MUST readvertise its
       own TIEs (since the new `PacketHeader` will contain a different
       level than before).  Sequence number of each TIE MUST be
       increased.

What if a "derived level" (vs. "defined level") changes?

What is the "level" in TIE PacketHeader for? What if it's only used in LIE?

   A node starting with LEVEL_VALUE being 0 (i.e. it assumes a leaf
   function or has a CONFIGURED_LEVEL of 0) MUST follow those additional
   procedures:

What's different between "it assumes a leaf function" and "has a CONFGURED_LEVEL of 0"? Perhaps just say "has a CONFIGURED_LEVEL of 0"?

-------------

   Certain use cases seem to necessitate distribution of essentialy KV
   information that is generated in the leafs in the northbound
   direction.  Such information is flooded in KV N-TIEs.  Since the
   originator of northbound KV is preserved during northbound flooding,	
   overlapping keys could be used.  However, to omit further protocol
   complexity, only the value of the key in TIE tie-broken in same
   fashion as southbound KV TIEs is used.

The last sentence of the above paragraph seems to read better as the following:

   ... only the value of the key is used in TIE tie-broken in same fashion
   as southbound KV TIEs.

4.3.7.  Leaf to Leaf Procedures
      ...
      never form southbound adjacencies.

Leaves never have southbound adjacencies (w/ or w/o leaf-to-leaf), right?

   ... Spine 22, having
   computed reachability to all prefixes in the network, advertises with
   the default route the ones that are reachable only via lower level
   neighbors that spine 21 does not show an adjacency to.

s/advertises with/advertises beside/

It's better to explicitly point out that this is the disaggregation case.

   The prefix 121 advertised by spine 22 does not have to be propagated
   further towards leafs since they do no benefit from this information.
   Hence the amount of flooding is restricted to spine 21 reissuing its
   S-TIEs and reflection of those by node 111 and node 112.

Do you mean "spine 22 reissuing its S-TIE" (not spine 21)?
Why would node 111/112 reflect the disaggregation routes?
There is text talking about disaggregation routes are not reflected.

5.3.  Partitioned Fabric

The section title says "partitioned fabric", but the fabric did not really partition, right?

   based on Section 4.3.12 it may advertise its loopback address as
   south PGP to remain reachable "from the south" for operational
   purposes.  This is necessary since A02 will NOT route towards A01
   using the E-W link (doing otherwise may form routing loops).

What about reachability from nodes in other PoDs, i.e., packets need to come down via a spine?
How does A02 route to A01? Is it that the south PGP learnt via the E-W link will be used?

Thanks!
Jeffrey

---

From: Tony Przygienda <[email-masked]>
Date: Tue, 1 May 2018 11:35:51 -0700

inline, we can chew more details in the interim tommorow, I plan to have a
picture with ZTP & how it hangs together with LIEs (since it's integrated
basically on LIEs) ...

> - nodes right below the spine must have pod configured; lower nodes/leaves

correct, the PoD terminology is not fully gelled yet, "undefined" also
means "any", i.e. value 0 which is bit confusing but I have no better
terms.

>
> On the contrary, w/ ZTP, leaves must be configured with level 0
> (explicitly or implicitly via leaf_only or l2l flag)?
>

that is incorrect. w/ZTP the leafs _may_ be configured, if they are with
either 0/leaf_only or l2l they _will_ stick to be 0 & leaf, if not they
just end up @ lowest level whatever it is (and it can be even different
height per PoD), I will paste couple examples in for tommorow.

>             every pair of neighboring routers become adjacent.

yes, conciously. we really just have a p2p adjacency/neighbor which is the
same for RIFT. OSPF "adjacency" concept is quite difficult to implement and
really just an optimization for large number of parallel links.  Nothing
prevents a RIFT implementation to be "smart" in flooding about parallel
links but each link is really a separate adjacency.

That should be probably clarified a bit in draft for the OSPF heavy folks,
someone suggests a language ...

>
> The three-way handshake establishes neighborship; after that, level/pod is
> derived and then based on that, we determine if adjacency should be brought
> up.
>
> Don't understand the text in the last sentence above. The spines don't
> have PoD configured (they connect to multiple PoDs), right? If we change
> "the top spines" to "the top node right below a spine" then it makes sense.
>

would need pictures, basically PoD membership is "defined" in the top
spines of a PoD and "diffuses southbound".  The superspines (connecting
PoDs) are "undefined" or "any" so they will form adjacency to all PoD
values below but nothing diffuses.

> A spine node does not have north adjacencies. Do you mean "a middle node"

yes, definition needs improving, Spine should be "superspine" here

Spine:  The set of nodes that provide inter-PoD communication.  These
      nodes are also organized into levels (typically one, three, or
      five levels).  Spine nodes do not belong to any PoD and are
      assigned the PoD value 0 to indicate this.

and middle level is "spine" or "aggregation"

will come in -02

> Why don't S-TIEs contain local prefixes? Wouldn't that lead to better
> routes on its southern neighbors to those local prefixes?
>

yepp, correct observation. Important especially for loopbacks. In fact the
draft does imply that in other writedowns but this
paragraph does not. in -02

>           (Node122, layer 1, cost 1)))

yes, draft tried to be super-consistent with level vs. layer but something
always sneaks in ...

> Node 121: Node S-TIEs do not have link info shown. Better be consistent.

suggest diffs. I masaged that too many times to bother again ;-)

> What's different between NodeElement and NodeLinkElement?

none, artefact from previous tinkering, corrected ... This is just a
conceptual example anyway while schema is normative

> Need to spell out the details for unnumbered IPv4 interface case instead

yes, appendices with LIE & flooding FSM. Going into details here would
crowd the
draft with well-known things (for the link-state folks).

> No matter where the packet is received from, the rest of the text applies.

I have to think whether this is completely true given different prefix
types and their preference. I think it is.

>    hop variety since South Node TIEs are not re-flooded southbound

corrected Node S-TIEs

I'm apprehensive introducing the binding between SPFs and which direction
traffic is going, I don't think
it's necessary.

>    confirm bidirectional connectivity.

ok, I clarify, what you _really_ do is the following

"

N-SPF uses northbound and East-West adjacencies in the computing node's
node N-TIEs (since if the node is a leaf it may not have generated a node
S-TIE) when starting Dijkstra. Observe that N-SPF is really just a one hop
variety since Node S-TIEs are not re-flooded southbound beyond a single
level (or East-West) and with that the computation cannot progress beyond
adjacent nodes.

Once progressing, we are using the next level's node S-TIEs to find
according adjacencies to verify backlink connectivity. Just as in case of
IS-IS or OSPF, two unidirectional links are associated together to confirm
bidirectional connectivi

"

>
> 4.2.5.2.  Southbound SPF
>
>    S-SPF uses only the southb

ound adjacencies in the south node TIEs,  <-- "Node N-TIEs"?

nope, node S-TIEs.  The SPF is counter-intuitive in this respect, you start
own node with the direction of the SPF, then you use counter-direction to
verify backlinks and then you take the next node's TIEs in the SPF
direction again ...

> "north/south node TIE" is ambiguous in that it could be interpreted as
> "north/south node's TIE" (TIE of the north/south node), hence the suggested
> change to be stringent.
>

you use both node TIE types as I write ... One to progress, other to
validate backlinks.

> "its default route encompasses" is unnecessary? Black-holing or
> sub-optimal forwarding is not caused by this node's default route. Rather,
> it is caused by the default route of the other nodes that has no possible
> next-hops below. Even if this nodes does not have a default route, the
> problem still exist and the solution is the same.
>

hmmm, yes, this is correct but we do NOT see the "other node's" default
routes in the same level & hence we check against the disaggregating node's
one.

rest later, need prepare interim material now ...

--- tony

---

From: Jeff Tantsura <[email-masked]>
Date: Tue, 1 May 2018 12:38:18 -0700

Very happy to see further clarifications and increasingly better readability of the draft!

Given - not all the comments will make it to the updated draft, I think WiKi alike format would be rather useful.

Wdyt?

Regards,
Jeff

> On May 1, 2018, at 11:35, Tony Przygienda <[email-masked]> wrote:
> 
> inline, we can chew more details in the interim tommorow, I plan to have a picture with ZTP & how it hangs together with LIEs (since it's integrated basically on LIEs) ... 
> 
>> On Tue, May 1, 2018 at 8:50 AM, Jeffrey (Zhaohui) Zhang <[email-masked]> wrote:
>> Hi Tony,
>> 
> correct, the PoD terminology is not fully gelled yet, "undefined" also means "any", i.e. value 0 which is bit confusing but I have no better terms. 
>  
>> 
> that is incorrect. w/ZTP the leafs _may_ be configured, if they are with either 0/leaf_only or l2l they _will_ stick to be 0 & leaf, if not they just end up @ lowest level whatever it is (and it can be even different height per PoD), I will paste couple examples in for tommorow. 
>  
>> 
> yes, conciously. we really just have a p2p adjacency/neighbor which is the same for RIFT. OSPF "adjacency" concept is quite difficult to implement and really just an optimization for large number of parallel links.  Nothing prevents a RIFT implementation to be "smart" in flooding about parallel links but each link is really a separate adjacency. 
> 
> That should be probably clarified a bit in draft for the OSPF heavy folks, someone suggests a language ...  
>  
>> 
> would need pictures, basically PoD membership is "defined" in the top spines of a PoD and "diffuses southbound".  The superspines (connecting PoDs) are "undefined" or "any" so they will form adjacency to all PoD values below but nothing diffuses. 
>  
>> 
> yepp, correct observation. Important especially for loopbacks. In fact the draft does imply that in other writedowns but this 
> yes, draft tried to be super-consistent with level vs. layer but something always sneaks in ... 
>  
>> 
> none, artefact from previous tinkering, corrected .... This is just a conceptual example anyway while schema is normative
>  
>> 
> yes, appendices with LIE & flooding FSM. Going into details here would crowd the 
> I have to think whether this is completely true given different prefix types and their preference. I think it is. 
>  
>> 
> I'm apprehensive introducing the binding between SPFs and which direction traffic is going, I don't think
> it's necessary. 
> 
>  
> N-SPF uses northbound and East-West adjacencies in the computing node's node N-TIEs (since if the node is a leaf it may not have generated a node S-TIE) when starting Dijkstra.. Observe that N-SPF is really just a one hop variety since Node S-TIEs are not re-flooded southbound beyond a single level (or East-West) and with that the computation cannot progress beyond adjacent nodes.
> 
> Once progressing, we are using the next level's node S-TIEs to find according adjacencies to verify backlink connectivity. Just as in case of IS-IS or OSPF, two unidirectional links are associated together to confirm bidirectional connectivi
> 
> 
> "
>  
>> 
>> 4.2.5.2.  Southbound SPF
>> 
> nope, node S-TIEs.  The SPF is counter-intuitive in this respect, you start own node with the direction of the SPF, then you use counter-direction to verify backlinks and then you take the next node's TIEs in the SPF direction again ... 
>  
> you use both node TIE types as I write ... One to progress, other to validate backlinks. 
>  
>> 
>>        since their scope is controlled by configuration..
>> 
> hmmm, yes, this is correct but we do NOT see the "other node's" default routes in the same level & hence we check against the disaggregating node's one. 
>  
> 
> _______________________________________________
> Rift mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/rift