Subject: Re: [Idr] Feedback on 5575-bis

From: Christoph Loibl <[email-masked]>
Date: Thu, 4 Oct 2018 12:19:27 +0200

Hi Jeff, 

Thank you very much for your detailed feedback on the draft.

As I promised this is the full report on your feedback on 5575bis draft. I appreciate discussion on the list. And will update the draft document next week according to the discussion.

However, I really need to make clear (again): With this -BIS we want to make things clearer and want to improve interoperability. We avoid fundamental changes that lead to incompatibility (esp. when it comes to encoding of the NLRI and propagation of the NLRI). Please always keep that in mind when suggesting changes (basically this is *not* flowspec 2.0)

Said that, I think the current RFC5575 implementations (at least those I am aware of) are widely compatible/compliant implementations also of the 5575BIS but may lack some minor details that have previously not been specified to that granularity (in RFC5575).

See my comments inline (CL).

Cheers Christoph

> From: Jeffrey Haas <[email-masked] <mailto:[email-masked]>>
> Subject: [Idr] Feedback on 5575-bis
> Date: 12. July 2018 at 20:58:55 CEST
> To: [email-masked] <mailto:[email-masked]>
> 
> As I promised Sue, I've done a thorough re-read of draft-ietf-idr-rfc5575bis.
> This re-read was partially prompted by recent work in my employer's
> implementation so this was very fresh.
> 
> Here are my comments:
> 
> :   A Flow Specification received from an external autonomous system will
> :   need to be validated against unicast routing before being accepted.
> :   If the aggregate traffic flow defined by the unicast destination
> :   prefix is forwarded to a given BGP peer, then the local system can
> :   safely install more specific flow rules that may result in different
> :   forwarding behavior, as requested by this system.
> 
> Issue: Redirections may need additional scrutiny.  While the destination
> based check on the received unicast routing is good, it's still possible for
> the external AS to influence forwarding beyond "rate-limit/drop" if scrutiny
> is not given to the other control communities.  This issue is compounded
> with the redirect-ip feature, not in this document.
> 
> I'd suggest this be mentioned in an operational security considerations
> section.

CL: As a matter of fact there is already a paragraph mentioning this particular problem. However I suggest to make it clearer with a operational security considerations section as you suggested:

   While the general verification of the traffic filter NLRI is
   specified in this document (Section 6) the traffic filtering actions
   received by a third party may need custom verification or filtering.
   In particular all non traffic-rate actions may allow a third party to
   modify packet forwarding properties and potentially gain access to
   other routing-tables/VPNs or undesired queues.  This can be avoided
   by proper filtering of action communities at network borders and by
   mapping user-defined communities (see Section 7) to expose certain
   forwarding properties to third parties.

> 
> :   In many deployments of BGP Flow Specification, the Flow Specification
> :   information has replace existing host length route advertisements.
> 
> s/replace/replaced/.  Also, it's unclear in the context of this paragraph
> why this is interesting.  A bit of explanatory text explaining the use of
> such most-specific routes for redirecting traffic would be useful.

CL: Suggest to remove the sentence from the draft

> 
> :   Experience with previous BGP extensions has also shown that the
> :   maximum capacity of BGP speakers has been gradually increased
> :   according to expected loads.  Taking into account Internet unicast
> :   routing as well as additional applications as they gain popularity.
> 
> The last sentence is dangling.

CL: Suggest to rephrase:

   Experience with previous BGP extensions has also shown that the
   maximum capacity of BGP speakers has been gradually increased
   according to expected loads.  For example Internet unicast routing as
   well as other BGP applications increased their maximum capacity as
   they gain popularity.

> :   Flow Specification components must follow strict type ordering by
> :   increasing numerical order.  A given component type may (exactly
> :   once) or may not be present in the specification.  If present, it
> :   MUST precede any component of higher numeric type value.
> 
> .... and what if they don't?  Is it malformed?  Should the implementation
> attempt to re-order?

CL: [1] I sent an email roughly a month ago to start a discussion about the encoding of the NLRI - with no response:

*) NLRI is widely ment opaque to BGP (no changes to RFC5575)

but: 

*) BGP needs to decode the NLRI to the first component (DST-IP) for the verification (Section 6) (not more of it! - the rest is part of the application that receives the NLRI)

Since changing the above properties would introduce a fundamental change to FS I strongly beleave that it does not fit into a 5575bis.

If the components are not in the right order the encoding _is_wrong_. However BGP cannot detect this, because its knowledge about the correct encoding ends after the very first component (DST-IP). So, to obey the rules (RFC5575) of an opaque NLRI FS-garbage gets propagated. Only when the decoding of the NLRI actually happens (not in BGP) such invalids can be detected and need to be ignored. 

Additionally reordering is not possible because BGP has no understanding of the components (which is actually good because it allows future extensions) - (btw. reordering/re-encoding of the NLRI is potentially dangerous because it may produce “ghost”/“mutated" duplicates of an NLRI in the network).

I suggest to add a paragraph to the draft to make it clear.

Please also refer to my Email one or two months ago - the suggested changes (100% compatible with current implementations) may also help to introduce future extensions.

> :   All combinations of component types within a single NLRI are allowed,
> :   even if the combination makes no sense from a semantical perspective.
> :   If a given component type within a prefix in unknown, the prefix in
> :   question cannot be used for traffic filtering purposes by the
> :   receiver.  Since a Flow Specification has the semantics of a logical
> :   AND of all components, if a component is FALSE, by definition it
> :   cannot be applied.  However, for the purposes of BGP route
> :   propagation, this prefix should still be transmitted since BGP route
> :   distribution is independent on NLRI semantics.
> 
> The issue here is when we encounter an unknown component, we can no longer
> verify the syntactic correctness of the NLRI.  This means we will pass
> potentially invalid entries downstream.  Downstream receivers must thus not
> drop the session, but should use treat-as-withdraw semantics.
> 
> This issue in validation, as I tersely noted in a prior IDR thread, really
> needs discussion in the document.  It impacts the ability to verify, to add
> new component types, and potentially cause invalid NLRI to propagate
> downstream.

CL: See the above comment. As long as BGP treats the NLRI as an opaque value we do not have a problem with all this (not even with new FS components). Downstram propagation of the garbage can be partly avoided (see my Email on that topic) at some cost. But this issue exists with every BGP opaque NLRI - this is not special to FS.

> :   The <type, value> encoding is chosen in order to allow for future
> :   extensibility.
> 
> It should be mentioned that due to the property above that extensions are
> problematic.

CL: This is true, and I think that should be removed. (See also the 2 comments above)

> 
> 4.2.3:
> :     0   1   2   3   4   5   6   7
> :   +---+---+---+---+---+---+---+---+
> :   | e | a |  len  | 0 |lt |gt |eq |
> :   +---+---+---+---+---+---+---+---+
> :
> :        Numeric operator
> 
> The 0 field is unclear what the expected behavior is when it's NOT zero.
> 
> :      a - AND bit.  If unset, the previous term is logically ORed with
> :      the current one.  If set, the operation is a logical AND.  It
> :      should be unset in the first operator byte of a sequence.  The AND
> :      operator has higher priority than OR for the purposes of
> :      evaluating logical expressions.
> 
> It's not clear what should be done if AND is set in the first list entry.
> 
> :      len - length of the value field for this operand encodes 1 (00) -
> :      4 (11) bytes.  Type 3 flow component values are always encoded as
> :      single byte (len = 00).
> 
> "always encoded" begs the question about what to do when it isn't.  Being
> encoded as 4 bytes of length but a length of 1 is still correct, e.g.  Does
> Postel's maxim apply here?

CL: I suggest change all the occurrences of 0, and must be 0 to “MUST be encoded as 0 and ignored on decoding”. Also we suggest (SHOULD) to encode the length of the compoents. Furthermore we suggest a-and bit MUST be unset on first operator, and ignored on decode (mainly for compatibility reasons - I do not want to discuss if Postel was right or not :-) ):

      e - end-of-list bit.  Set in the last {op, value} pair in the
      list.

      a - AND bit.  If unset, the previous term is logically ORed with
      the current one.  If set, the operation is a logical AND.  In the
      first operator byte of a sequence it MUST be encoded as unset and
      and MUST be treated as always unset on decoding.  The AND operator
      has higher priority than OR for the purposes of evaluating logical
      expressions.

      len - length of the value field for this operand encodes 1 (00) -
      4 (11) bytes.  Type 3 flow component values SHOULD be encoded as
      single byte (len = 00).

      0 - MUST be set to 0 on NLRI encoding, and MUST be ignored during
      decoding

      lt - less than comparison between data and value.

      gt - greater than comparison between data and value.

      eq - equality between data and value.

> 
> 4.2.4
> :      Defines a list of {operator, value} pairs that matches source OR
> :      destination TCP/UDP ports.  This list is encoded using the numeric
> :      operator format defined in Section 4.2.3.  Values are encoded as
> :      1- or 2-byte quantities.
> 
> It's not clear what to do if the encoding is > 2 byte.  Ditto for sections
> 4.2.5, 4.2.6, etc.

CL: Same solution as the above. (4.2.3)

> 4.2.9 has a similar "must be zero" set of bits that need to have their
> behavior specified.

CL: Same solution as above (4.2.3)

> 
> 4.2.11
> :      Defines a list of {operator, value} pairs used to match the 6-bit
> :      DSCP field [RFC2474].  This list is encoded using the numeric
> :      operator format defined in Section 4.2.3.  Values are encoded
> :      using a single byte.  The two most significant bits are zero and
> :      the six least significant bits contain the DSCP value.
> 
> It's not specified what to do when the top two bits are set.

CL: Same solution as above (4.2.3)

> 
> 4.2.12
> :    +---+---+---+---+---+---+---+---+
> :    |   Reserved    |LF |FF |IsF|DF |
> :    +---+---+---+---+---+---+---+---+
> :
> :      Bitmask values:
> :
> :         Bit 7 - Don't fragment (DF)
> :
> :         Bit 6 - Is a fragment (IsF)
> :
> :         Bit 5 - First fragment (FF)
> :
> :         Bit 4 - Last fragment (LF)
> 
> 
> While the general behavior of unexpected lengths is already unclear, it's
> even less clear here.  
> 
> Reserved behavior is not defined.  "Must be zero on transmit, should be zero
> on receive?"
> 
> It's also not specified whether this is "any of" or not.  This impacts
> whether multiple bits being set is valid for some combinations.

CL: Regarding the reserved bits: Same solution as above (4.2.3).

Regarding the any-of or distinct/bitwise match. This is explained in section 4.2.9 where the bit-operator is introduced.

   m -   Match bit.  If set, this is a bitwise match operation defined
      as "(data AND value) == value"; if unset, (data AND value)
      evaluates to TRUE if any of the bits in the value mask are set in
      the data

> 
> 5.
> 
> :   filtering (prevention of traffic-based, denial-of-service (DOS)
> :   attacks, traffic filtering in the context of BGP/MPLS VPN service,
> :   and centralized traffic control for SDN/NFV networks) requires
> 
> The separate clauses here likely need to be semicolon separated since the
> first clause is a compound separated by a comma.  Alternatively leave to the
> RFC editor to straight it out.

CL: Editorial changes.

> :   Section 8 has details on the limitation
> :   of previous mechanisms and why BGP Flow Specification version 1
> :   provides a solution for to prevent DOS and aid BGP/MPLS VPN filtering
> :   rules.
> 
> I think this is the first time in the document this is referred to as v1 of
> the spec.

CL: This is definitely wrong and all occurrences of FS v1 need to be removed.

> 
> :   This Flow Specification NLRI defined above to convey information
> :   about traffic filtering rules for traffic that should be discarded or
> :   handled in manner specified by a set of pre-defined actions (which
> :   are defined in BGP Extended Communities).  This mechanism is
> :   primarily designed to allow an upstream autonomous system to perform
> :   inbound filtering in their ingress routers of traffic that a given
> :   downstream AS wishes to drop.
> 
> There's several bits of awkwardness in the above paragraph.  But among them
> is the presumption of drop being the default choice in the last sentence.

CL: There is a default in the specification under section 7 (accept). This is again also a question of extensibility. If new action-communities are “invented” a older implementation may not be able to identify those extended communities and thus may think that *no* action is specified (so it is forwarding the traffic to a node that knows about that new action). 

However FS-filter matching normally stops at the first match (subsequent FS-filters after the match are not taken into consideration). 

I suggest to remove the invalid “default” choice from this paragraph and explain the behaviour as described above. Section 7 as you pointed out below however should be a bit more verbose on the matching process.

> 5.1
> :   The relative order of two Flow Specification rules is determined by
> :   comparing their respective components.  The algorithm starts by
> :   comparing the left-most components of the rules. 
> 
> The importance of whether the components are sorted becomes critical here.
> Given the semantic ambiguity of components the receiver is unable to decode
> (unknown), the length check also can't be applied here.  This is mostly
> important for likely buggy cases where two NLRI are semantically identical
> but mis-ordered.  Compare vs. the RFC 3107 withdraw ambiguities for label
> value some time ago.
> 
> A trite example is consider two NLRI both containing the type 4 port field,
> both containing the same 1-byte value (e.g. 255), but one encoded using a
> single byte of length, the other with two bytes of length.  Is this an
> implicit update or not?
> 
> Somewhat similarly, duplicate components aren't specifically called out, but
> potentially implied (as noted in an IDR exchange) by the requirements of the
> sort order.  Exactly what the behavior is in the face of duplicates should
> be specified.

CL: NLRI re-en-coding is not acceptable since it introduces a hole lot of problems. Opaqueness of the NLRI to BGP is somehow vital to avoid such situations altogether. See also the comment above [1].

> 6.
> :   The underlying concept is that the neighboring AS that advertises the
> :   best unicast route for a destination is allowed to advertise flow-
> :   spec information that conveys a more or equally specific destination
> :   prefix.  Thus, as long as there are no more specific unicast routes,
> :   received from a different neighboring AS, which would be affected by
> :   that filtering rule.
> 
> 
> Observation: An address space hijack that is not filtered, e.g.
> advertisement of a more specific route, may prevent inter-as flowspec from
> validating.  This might be worth putting in the operational considerations
> section.

CL: Yes, adding the operational considerations section:

   Since verfication of the traffic filtering NLRI is tied to the
   announcement of the best unicast route, a unfiltered address space
   hijack (e.g. advertisement of a more specific route) may cause this
   verification to fail and consequently prevent Flow Specification
   filters from being accepted by a peer.
> 
> 7.
> :   Multiple traffic actions may be present for a single NLRI.  The
> :   traffic actions are processed in ascending order of the sub-type
> :   found in the BGP Extended Communities.  If not all of them can be
> :   processed the filter SHALL NOT be applied at all (for example: if for
> :   a given flow there are the action communities rate-limit-bytes and
> :   traffic-marking attached, and the plattform does not support one of
> :   them also the other shall not be applied for that flow).
> 
> Typo on platform.
> The extended communities defined are not only a given sub-type, but also a
> type field.  The above suggestion is ambiguous about the case where the
> subtypes are equal but the types are distinct.
> 
> I'm also wondering if the "applied at all" is the desired wording.  Does
> that mean, as earlier presented in the draft, we get:
> 
> :   The default action for a traffic filtering Flow Specification is to
> :   accept IP traffic that matches that particular rule.
> 
> Or does it mean that the rule is ignored?  The implications are different: A
> pass permits traffic to pass whereas ignore is effectively a "delete rule",
> which may cause traffic that would otherwise pass the filter to be impacted
> differently by subsequent rules.

CL: You are right this is not very clear. I think that a rule where not all of the actions can be performed should generate a match (meaning that subsequent filters are not evaluated) and the packet is treated as default-accept.

> 7.1
> :   The remaining 4 octets carry the maximum rate information in IEEE
> :   floating point [IEEE.754.1985] format, units being bytes per second.
> :   A traffic-rate of 0 should result on all traffic for the particular
> :   flow to be discarded.
> 
> It is worth noting that while we could simply end up with preposterously
> large numbers, there is also the edge case that these floating point values
> may be NEGATIVE.  The draft may wish to note that negative numbers should be
> one of: ignored (see prior comment about how this impacts rule order), or
> treated as zero (discard)
> 

CL: I suggest to add a lower bound there:

   flow to be discarded.  On encoding the traffic-rate MUST NOT be
   negative.  On decoding negative values MUST be treated as zero
   (discard all traffic).

> 7.2
> See 7.1

CL: See 7.1 ;-)

> 7.4
> There's a level of ambiguity in what happens when there's more than one
> redirect extended community:
> 
> :   Interferes with: All other redirect functions.  All redirect                                     
> :   functions are mutually exclusive.  If this redirect function exists,
> :   then no other redirect functions can be processed.
> 
> We're allowed to install to any VRF:
> :   If several local instances match this criteria,
> :   the choice between them is a local matter
> 
> but this is partially also interacting with the text from earlier:
> :   Multiple traffic actions may be present for a single NLRI.  The
> :   traffic actions are processed in ascending order of the sub-type
> :   found in the BGP Extended Communities.  If not all of them can be
> :   processed the filter SHALL NOT be applied at all
> 
> So, consider the case where we have two redirect extended communities.  If
> one of them does not have any matching VRFs, should the "SHALL NOT be
> applied at all" kick in?  If both can be used, do we pick the first as
> implied by ascending order?  Or do we take a wider reading of the 7.4 text
> and let the implementation just pick one?

CL: As of section 7.6. it is not allowed to have multiple redirect extended communities (-> interfering actions -> treat as withdraw):

   1.  Redirect-action-communities (0x8008, 0x8108, 0x8208):

       The three redirect-communities are mutually exclusive.  Only a
       single redirect community may be associated with a Flow
       Specification otherwise they are interfering.

> This indirectly raises a different point: For each of the action communities
> that aren't expected to allow multiple instances, how do you deal with the
> repeats?  E.g. two traffic-rate extended communities?

CL: As of section 7.6 this is not allowed:

   2.  All traffic-action communities (including redirect-actions):

       Multiple occurences of the same (sub-type and type) traffic-
       action associated with a Flow Specification are always
       interfering.
See also the above

> 7.6 covers the fact that only a single redirect community can be associated
> with a flowspec.  However, given that some routers may have target:A working
> but not target:B and vice versa elsewhere, we shouldn't necessarily restrict
> this.  
> 

CL: yes this is a known drawback if we limit the redirect actions. We may also go with sorting them and picking the lowest/highest possible.

> 7.6.1 illustrates the idea that they are a conflict that should be treated
> as withdraw.  I believe this is not the desired behavior.

CL: as above in the specification it actually is. But as you pointed out maybe it is better to loosen this restriction and get some ordering in place (what should happen if a device is able to follow 2 redirect rules? We should not duplicate the packet :-) )  

> 
> 9.2 Another reference to this being version 1.

CL: remove.
> 13.
> s/MuPedro/Pedro/

CL: apply regex/replace

> — 

Christoph 

-- 
Christoph Loibl
[email-masked] <mailto:[email-masked]> | CL8-RIPE | PGP-Key-ID: 0x4B2C0055 | http://www.nextlayer.at <http://www.nextlayer.at/>

---

From: Jeffrey Haas <[email-masked]>
Date: Thu, 4 Oct 2018 12:27:37 -0400

Christoph,

> However, I really need to make clear (again): With this -BIS we want to make things clearer and want to improve interoperability. We avoid fundamental changes that lead to incompatibility (esp. whe…

[Some moralization about -bis documents.]

-bis documents are usually issued to clarify or fix broken behaviors.  While
this often means fixing easy errata, it also means the opportunity to
address things fundamentally broken in the original specificiation.

While I realize the authors' intent may be to do the least work necessary,
changes are already present that impact previously on-the-wire valid
encoding in a way that is not necessarily fully backward compatible.  And
*that's fine*, as long as it has working group consensus.

(Specific example being restriction of various filters to be of a specific
field length when that field length had a natural length, but wasn't
previously restricted against larger lengths in the original spec.)

So, while it may be the authors' intent to try to keep things small, it's
not out of order to try to make sure important things are addressed.  

>    forwarding properties to third parties.

This is reasonable text.

> CL: Suggest to remove the sentence from the draft

Works for me.

> CL: [1] I sent an email roughly a month ago to start a discussion about the encoding of the NLRI - with no response:

You may have noticed I and others have erratic IETF reply time. :-)

> If the components are not in the right order the encoding _is_wrong_. However BGP cannot detect this, because its knowledge about the correct encoding ends after the very first component (DST-IP).…

The "opaque" requirement is the thing I fundamentally question and I believe
is deserving of more WG discussion.  As you note in your own paper on
flowspec given at RIPE 74:

"In that case such a BGP UPDATE may lead to BGP sessions flapping not only
on adjacent routers but also on remote routesrs not directly connected with
the router that originates that NLRI as long as it is part of that same
flow-specification domain".

We fall back to an NLRI form of what I have called "optional transitive
nonsense" when discussing such errors for Path Attributes.  

Don't understand it?  Propogate it.  Cause issues downstream.

As you note in that same paper, at least one vendor fails to propagate
flowspec routes when they don't install a filter.

My employer's implementation currently drops sessions when components that
aren't understood are received.  This is arguably wrong, but 5575 existed
before RFC 7606 error handling procedures became the norm.  These days,
"treat as withdraw" may be the best behavior if not propagated opaquely.

I'll note that your paper didn't test for this condition.  Finding out what
the other implementations do is reasonable in a -bis situation to determine
if this "propagate opaque" is the best idea.

> Additionally reordering is not possible because BGP has no understanding of the components (which is actually good because it allows future extensions) - (btw. reordering/re-encoding of the NLRI is…

This is already problematic.  Consider the case of L3VPNs where the label
portion of the NLRI wasn't cleanly specified.  A result of this was edge
cases of key comparison being done differently on some implementations.

Basically, while I agree with the core point, BGP extensions already force
us to consider such things.

> Please also refer to my Email one or two months ago - the suggested changes (100% compatible with current implementations) may also help to introduce future extensions.

URL, please?

> CL: See the above comment. As long as BGP treats the NLRI as an opaque value we do not have a problem with all this (not even with new FS components). Downstram propagation of the garbage can be pa…

NLRI is generally NOT opaque in BGP.  This is a fundamental design issue of
the protocol that prevents the deployment of completely generic propogation
infrastructure.

What did you have in mind here?

[...]
> CL: I suggest change all the occurrences of 0, and must be 0 to “MUST be encoded as 0 and ignored on decoding”. Also we suggest (SHOULD) to encode the length of the compoents. Furthermore we sugges…

This is a point where a blanket declaration risks making existing
implementations non-compliant.  Again, poentially in scope, but not quite
consistent with your stated aims. :-)

>       has higher priority than OR for the purposes of evaluating logical

While I think this is reasonable, it does run up against some issues with
regards to encoding.  As you note, how we treat issues for reordering apply
here.

This is because implementations that follow this behavior are likely to mask
off the and behavior.  Implementations often regenerate the NLRI from
internal data structures.  As a result, doing the right thing runs into one
of your other concerns.

IMO, the right thing to do is keep the text above.

>       4 (11) bytes.  Type 3 flow component values SHOULD be encoded as

I'm good with this.  Same to other length comments later on.

>       0 - MUST be set to 0 on NLRI encoding, and MUST be ignored during

SHOULD is more appropriate here, lest you break future extensions.

> CL: Same solution as above (4.2.3)

SHOULD be zero and ignored on receipt is fine.

>       evaluates to TRUE if any of the bits in the value mask are set in

Agreed.

> I suggest to remove the invalid “default” choice from this paragraph and explain the behaviour as described above. Section 7 as you pointed out below however should be a bit more verbose on the mat…

I'll look for the proposed text.

[...]
> CL: NLRI re-en-coding is not acceptable since it introduces a hole lot of problems. Opaqueness of the NLRI to BGP is somehow vital to avoid such situations altogether. See also the comment above [1…

See again prior comments.  Many implementations build data structurres from
received NLRI.  If you understand it, you may regenerate it in canonical
order.

If you don't understand it, your implementation has no choice to treat it
opaquely.

The issue here is cases where you do understand it, but it's inconsistent.
So, I don't think your general opaque comment holds here.

Our implementation is likely at some future point to permit propagation of
opaque entries, but will likely default to "treat as withdraw".

>    filters from being accepted by a peer.

Thanks.

> CL: You are right this is not very clear. I think that a rule where not all of the actions can be performed should generate a match (meaning that subsequent filters are not evaluated) and the packe…

I think that default accept is probably the SAFEST behavior, but is perhaps
not what is desired.  I suggest flagging this issue for explicit WG
discussion.

>    negative.  On decoding negative values MUST be treated as zero

This seems reasonable.

> CL: As of section 7.6. it is not allowed to have multiple redirect extended communities (-> interfering actions -> treat as withdraw):

Again, that's a change of behavior and one that interferes with scenarios
where flowspec routes may be propagated among a set of routers with disjoint
VRFs.

> CL: yes this is a known drawback if we limit the redirect actions. We may also go with sorting them and picking the lowest/highest possible.

This is a change of behavior and one I disagree with. :-)  Also, one that
I'll fight all the way through WGLC.  This is deployed behavior among my
customer-base.

That said, my request is that the spec discuss inconsistencies rather than
trying to preclude it.

It's also worth noting that "magic community" issues are endemic to BESS
documents.  I've been wanting to write a draft there discussing it, but it's
been on my low priority queue.

-- Jeff

---

From: Christoph Loibl <[email-masked]>
Date: Fri, 5 Oct 2018 11:11:40 +0200

Hi Jeff,

> On 04.10.2018, at 18:27, Jeffrey Haas <[email-masked]> wrote:
> 
> Christoph,
> 
>> […]

I appreciate it very much, that this is moving again :-)

From your reply I distilled the 3 major points (to be honest those were no surprises to me) for additional discussion (not that we should forget about the others and comments are very welcome!):

1) What does BGP know about FS NLRI encoding “opaqueness”:

Most of your arguments lead back to the “opaqueness” property described in RFC5575. I am not so much in favour of propagating garbage. However, changing this behaviour (as specified in RFC5575) was not an action-point that I (I cannot speak for the other authors) considered very useful for better interop. I understand that currently, some implementations do not obey the RFC5575 specification in that particular manner. As a network operator however, I am not so much concerned about this as long as it serves our needs from an operational standpoint (gives us the flexibility and stability that we need - It currently does not, but maybe not because of this particular part of the specification), but I understand that implementation-wise this could be a completely different story.

Do we have better chances that implementations will follow the new specification (and thus increased interoperability + stability) if we discard the “opaqueness” stuff altogether? And, does this lead to a better (less complex, better scalable, …) specification and implementation? If *yes* I am totally in favour of going into that direction. It needs quite a lot changes (which I think, to a large percentage, do not badly break things out there) and some discussion:

*) Reencoding “ghosts”/“duplicates":

The complexity of the NLRI may allow to encode the same thing in multiple ways (I think you pointed that out already). Consider the numeric-operator. For example you may have a comparison for a tcp-port number represented as “=3, =4, =5” or “=4 =5 =3” (this can/should be fixed by having the implementations sort within the components by the numeric-values). However, when it comes to components based on the binary-operator I am not sure if sorting the “bitmasks” is in particular useful (if possible at all - this needs some more thinking). So we may still get some inconsistencies on recoding. - I will look into this next week.

BTW: I am not even sure if recoding the NLRI (even though it creates different versions of the same thing) is a big problem at all. Since we are propagating along a tree (ie AS-path or other intra-AS mechanisms for routing information loop-avoidence).

*) FS Extentions / Gracefully adding new component-types to the NLRI:

Option 1: Treat all NLRIs with unknown components as withdraw
Option 2: specify a component structure for future components that allows unknown components to be dissected/re-encoded and propagated (if desired). [1]
Option 3-: … you name it.
 
[1] - here the URL to my email I referred quite a few times already: http://ietf.10.n7.nabble.com/Feedback-on-5575-bis-tt568777.html#a570460 <http://ietf.10.n7.nabble.com/Feedback-on-5575-bis-tt568777.html#a570460> - please look into paragraph =2= if the mail (the extensibility) - the rest of it has already been brought in this discussion.

*) Propagation of unknown component-types:

In the opaqueness case there cannot be a discussion about this because BGP does not know anything about the components and is propagating all the garbage. If BGP has full understanding of the NLRI - what should happen in case of unknown NLRIs occurring in a BGP message: -> I vote for propagating (but never use it for filtering), but in this case we need to know how do decode unknown components (see above).

2) Interfering Traffic-Actions:

The idea here was to ignore what is not clear. - One could argue that in case of a rate-limit the lowest/highest should win (which?). Or in case of multiple redirect the one with the lowest/highes route-target on that particular node should win. To be honest (from an operator perspective) I am fine with any. We take care about not announcing stupid actions to the network anyway. I agree, that in some cases even multiple redirects can make sense (if finally only one is performed - duplication of the same data stream may not be what is wanted?). What do you + WG think is appropriate. Some simple numeric sorting comes into mind (but as a operator this may mean that you need to assign your route-targets according your preferred FS redirection). - RFC5575 does not specify this at all.

As an operator allowing multiple redirects associated with the same flow filter may arguable seem to be fancy feature, but I would really refrain using that in a production environment. Using my own set of BGP communities to convey that information and rewriting to whatever action is needed at network areas/nodes seems much more flexible to me than depending on a fixed algorithm (policy) that selects redirect targets on my behalf.

3) Default behaviour in case of a flow-match but not *all* actions can be “executed”:

Given the above example (allowing multiple redirects, multiple rate-limits, remarkings, … maybe some other future actions yet unknown) a match always requiring *all* actions to be executed or non (+ being default accept) seems not the right choice. I think that, if traffic actions are not required to be unique and we want some sorting/preference specified (see 2)  we need to come up with something different.

I will upload a new version repairing the minor issues you pointed out (see the previous Email in this thread -  if I see no objections on the list). At the same time I am not changing any of the other aspects (discussed in this Email) until we have a rough idea where we are heading for.

Cheers Christoph

---

From: Jeffrey Haas <[email-masked]>
Date: Fri, 5 Oct 2018 07:50:22 -0400

Christoph,

[Summary elided]

You captured the critical points. Thanks.

> On Oct 5, 2018, at 5:11 AM, Christoph Loibl <[email-masked]> wrote:

This is all I would ask for.  I don't expect one person's input to be sufficient consensus to do the other changes.  But the point of the IETF consensus process is that substantive technical issues are enough reasons for the chairs to hold up things until they're addressed.

Perhaps this might be a topic for the interim.

-- Jeff

---

From: Christoph Loibl <[email-masked]>
Date: Tue, 9 Oct 2018 15:38:33 +0200

Hi Jeff, IDR,

I am trying to write a summary on FS propagation in a non-opaque way (=BGP fully understanding of the NLRI + regenerating the byte-string) - Jeff: as you pointed out this is already the case with some of the implementations. I just want to better understand what we need to think of in case we want to go into that direction. (Questions at the end)

In one of my last emails I wrote that I think re-encoding of the FS NLRI may be dangerous because it may result in a different NLRIs getting created and propagated (see below). For example a simple tcp-port filter (port 137,138) can be encoded as: 

port == 137 || port == 138
port == 138 || port == 137
port >= 137 && port <= 138
…

The NLRI itself can express the same filter in multiple ways (I doubt, that we can change the specification to have a 1:1 mapping filter to NLRI without throwing the hole thing away - which I think is not at all desirable). 
	
> received NLRI.  If you understand it, you may regenerate it in canonical

I agree. Actually, I think that some of the behaviour that we observed in the lab years ago can be explained very well by this.

My questions here: 
------------------------

*) If the NLRI itself can express the same filter in multiple ways (see above). May regenerating the NLRI from a data structure produce a different NLRI byte-string (“ghost”)?

(I would say yes? - potentially) 

*) What are the implications if “ghost” NLRIs get propagated?

The propagation of the NLRI follows a tree from neighbor to neighbor (loops getting resolved by As-path, cluster-lists, … = independently from the NLRI itself) … withdraw follows the same mechanisms. However BGP-speakers may receive different byte-string versions of the same NLRI and maybe even treat both as two different routes (maybe propagating 2 NLRIs instead of one)? Is this something, that can also happen with other NLRIs? Is this dangerous?

Cheers Christoph

-- 
Christoph Loibl
[email-masked] | CL8-RIPE | PGP-Key-ID: 0x4B2C0055 | http://www.nextlayer.at

---

From: Jeffrey Haas <[email-masked]>
Date: Tue, 9 Oct 2018 10:34:10 -0400

Christoph,

On Tue, Oct 09, 2018 at 03:38:33PM +0200, Christoph Loibl wrote:
> I am trying to write a summary on FS propagation in a non-opaque way (=BGP
> fully understanding of the NLRI + regenerating the byte-string) - Jeff: as
> you pointed out this is already the case with some of the implementations.

I actually don't know what other implementations do.  If you do, that's a
data point.  What I said is that a -bis is a fine time to update problematic
pieces of documents based on actual deployed implementations.

> In one of my last emails I wrote that I think re-encoding of the FS NLRI
> may be dangerous because it may result in a different NLRIs getting
> created and propagated (see below). For example a simple tcp-port filter
> (port 137,138) can be encoded as: 
> 

And matter of fact, our implementation will likely re-encode some of these
cases.  This is because port range sets are canonicalized into our internal
data structures in a way that is functionally equivalent but optimal for our
firewall implementation.  And since we generate our NLRI from our internal
data structures, what we propogate will be different in some cases than what
we receive.

This is not a problem.

NLRI exchange is a local matter between two BGP peers.  The only thing that
is important is that you withdraw what you send and that these are in
agreement.

[...]
> I agree. Actually, I think that some of the behaviour that we observed in
> the lab years ago can be explained very well by this.
> 
> My questions here: 
> ------------------------
> 
> *) If the NLRI itself can express the same filter in multiple ways (see
> above). May regenerating the NLRI from a data structure produce a
> different NLRI byte-string (“ghost”)?
> 

Absolutely.

> The propagation of the NLRI follows a tree from neighbor to neighbor
> (loops getting resolved by As-path, cluster-lists, … = independently from
> the NLRI itself) … withdraw follows the same mechanisms. However
> BGP-speakers may receive different byte-string versions of the same NLRI
> and maybe even treat both as two different routes (maybe propagating 2
> NLRIs instead of one)? Is this something, that can also happen with other
> NLRIs? Is this dangerous?

Given a network distributes sets of flowspec NLRI and implementations mutate
the NLRI in a way that is functionally equivalent.  Given a downstream
router that receives the same flowspec filter that has been mutated in
different ways in the upstream path.  Then a downstream router may receive
the same flowspec programming in more than one format and thus can't declare
them to be duplicates for route selection purposes.

The result of this would be multiple rules installed, functionally
equivalent, with different but likely compatible precedence.

It's my belief, but I haven't done formal proof for, that the rules would
always be compatibly ordered.  I believe this is because such
canonicalizations should only be done within a given component and impact
the list-based components.

Where we would have actual forwarding problems would be the case where
sub-components are dropped or appended to.

A related incidental problem is if some NLRI are simply not forwarded
because they're not able to be installed locally.  You saw this in your
paper.  Such things could result in incorrect firewall installation due to
rules not being present that might be a necessary pass/reject.

-- Jeff

---

From: Christoph Loibl <[email-masked]>
Date: Thu, 18 Oct 2018 10:36:37 +0200

Hi,

> On 05.10.2018, at 11:11, Christoph Loibl <[email-masked]> wrote:
> 

As promised, I uploaded a new version of draft-ietf-idr-rfc5575bis (-09) that addresses most of the minor issues pointed out by Jeffery in a way posted on the list earlier. Issues that seem to need more discussions have not been addressed by this new version (I may make a short presentation on that in the interim next week).

Please look into the changes on datatracker:

https://datatracker.ietf.org/doc/draft-ietf-idr-rfc5575bis/

Cheers Christoph

---

From: "Dongjie (Jimmy)" <[email-masked]>
Date: Tue, 23 Oct 2018 02:28:27 +0000

Hi Christoph,

We are building the agenda for the upcoming interim meeting.

Could you let us know whether you need a time slot in the interim to present the update on 5575-bis and discuss the remained issues?

Best regards,
Jie

From: Idr [mailto:[email-masked]] On Behalf Of Christoph Loibl
Sent: Thursday, October 18, 2018 4:37 PM
To: idr wg <[email-masked]>
Subject: Re: [Idr] Feedback on 5575-bis

Hi,

On 05.10.2018, at 11:11, Christoph Loibl <[email-masked]<mailto:[email-masked]>> wrote:

Cheers Christoph

--
Christoph Loibl
[email-masked]<mailto:[email-masked]> | CL8-RIPE | PGP-Key-ID: 0x4B2C0055 | http://www.nextlayer.at

---

From: Christoph Loibl <[email-masked]>
Date: Tue, 23 Oct 2018 08:58:21 +0200

Hi Jie,

I am sorry that my initial mail the other day did not reach you (I sent it to Sue and John - according to the mail Sue sent out about the interim) … Yes, I would like to present on the 5557-bis draft:

In order to progress the draft, I would like to present roughly 10 minutes on draft-ietf-idr-rfc5575bis-09 and two possible changes in the proposed behaviour that may need some more discussion on the working group:

	a) 

A property already in RFC 5575 that seemingly was ignored by pretty much all the implemenations “BGP opaque - NLRI)  

	b)

Less restrictive interaction of interfering flow actions.

Thanks for asking!

Cheers Christoph

> On 23.10.2018, at 04:28, Dongjie (Jimmy) <[email-masked]> wrote:
> 
> Hi Christoph,
>  
> On 05.10.2018, at 11:11, Christoph Loibl <[email-masked] <mailto:[email-masked]>> wrote:
>  
> https://datatracker.ietf.org/doc/draft-ietf-idr-rfc5575bis/ <https://datatracker.ietf.org/doc/draft-ietf-idr-rfc5575bis/>
>  
> Cheers Christoph
>  
> -- 
> Christoph Loibl

---

From: Jeffrey Haas <[email-masked]>
Date: Fri, 26 Oct 2018 05:19:33 -0400

> As promised, I uploaded a new version of draft-ietf-idr-rfc5575bis (-09) that addresses most of the minor issues pointed out by Jeffery in a way posted on the list earlier. Issues that seem to need…

And for the record, aside from the issues you hope to discuss at the
interim, I believe this addresses my comments to date.

-- Jeff