Subject: Re: [GROW] The new BMP drafts and new BMP functionality

From: Job Snijders <[email-masked]>
Date: Sun, 22 Apr 2018 00:28:18 +0000

Dear WG,

I’d like to encourage the authors and stakeholders of current BMP work to
assess the below feedback and get back to the working group.

Kind regards,

Job

On Mon, 26 Mar 2018 at 15:24, Henk Smit <[email-masked]> wrote:

>
> Hello all,
>
> I've read the 2 new BMP drafts.
> I don't like them very much.
> I think we can do better.
>
>
> What the new drafts propose:
> ----
> The new drafts introduce new ways to report routes from the Adj-RIB-Out
> and Loc-RIB to bmp-stations.
>
> The Adj-RIB-Out draft does that by using a flag from the BMP per-peer
> header's flag-field. This new flag indicates that a reported route is
> from the Adj-RIB-Out in stead of the Adj-RIB-In.
> I don't like this.
>
> The flags-field has only 8 bits. We were using 3 of them. This draft
> uses a 4th. And the Loc-RIB draft introduces yet another flag. We'll
> be running out of flags soon.
>
> The Loc-RIB draft introduces a new peer-type, the so-called
> "Loc-RIB Instance Peer". Note, we already had a "Local Instance Peer".
> I don't find this naming very clear. I also don't like mixing three
> different types of routes (In, Out, Loc) in the same message-type.
>
>
> What I propose:
> ----
> We have 256 message-types. We are using 7 of those now.
> I think it would be wiser to introduce a new message-type for
> reporting Adj-RIB-Out routes. That is 1) a bit cleaner, and 2) it
> allows us to use the flags in the flag-field for more important or
> more useful things.
>
> If we are introducing a new message-type for Adj-RIB-Out, we can just
> as well introduce a new message type for Loc-RIB routes. That makes
> the distinction between the 3 different types of routes a lot cleaner.
>
>
> What about the old route-monitoring message ?
> ----
> Modern protocols are developed using TLVs.
> BMP does have TLVs in some of its messages too (init, term, peer-up,
> etc).
> However, the most important message in BMP, the route-monitoring message
> is fixed format. I think that is a bug.
> The route-monitoring message consists of:
> 1) the small common BMP header
> 2) the per-peer header
> 3) the reported route(s), encapsulated in a BGP update-message.
>
> There is no flexibility here to add more information.
> That is a short-coming.
> The whole body of the message should be TLV-based.
> Therefor I propose we use the following message-format for the
> new route-report messages.
>
> 1) small common BMP header
> 2) the per-peer header
> 3) TLVs, these consist of:
>     a) zero or more informational TLVs.
>        these can include tags, reason why a route was dropped or
> accepted,
>        did this route win best-path computation ? is the route installed
> in
>        the GRT/RIB/RTM ? we can add any new information here, in the
> future,
>        by just defining a new TLV.
>     b) a TLV which holds the BGP update-message.
>
>
> This format should apply to all 3 new message-types, for Adj-RIB-In,
> Adj-RIB-Out and Loc-RIB.
>
>
> What is the impact of this change ?
> ----
> The BMP protocol remains largely the same.
> The common BMP header and the per-peer header stay the same.
> The init and termination messages stay the same.
> The peer-up and peer-down messages stay the same.
> These last 4 messages are already really TLV-based.
> Only the route-monitoring messages need to change.
>
> Sending and parsing of the route-monitoring is largely the same
> as it is today. The only difference is that a bunch of TLVs can
> be sent along with the BGP update message. Existing bmp-station
> implementations can be easily altered to accept the 3 new
> route-monitoring
> messages, and just skip the TLVs. That's less than a day's work.
> If a bmp-station implementation wants to know about the new TLVs, new
> code can be written and introduced to parse and process the extra
> information on a TLV-by-TLV bases. The BMP protocol is now extensible.
>
>
> Do we need to bump BMP to version 4 ?
> ----
> It might be nice to do this.
> But technically there is no need. Old bmp-station implementations will
> accept the 3 new BMP messages, and (hopefully) skip the message-types
> they
> don't know about. Routers can have a configuration knob to send only
> old-fashioned route-monitoring messages (type 0). Or they could send
> routes in the 3 new message-formats. This is straight-forward to
> implement.
>
>
> Flags in the flags-field in the per-peer header
> ----
> The flags-field in the per-peer header has 8 bits.
> Currently we use 3 of those.
> V-bit indicates the peer uses an IPv6 address.
> L-bit indicates post-policy
> A-bit indicates 2-byte as-path
>
> I'd like to use a few more bits.
>
> 1) We have a bit to indicate that a route is sent with pre-policy
> attributes
>     or with post-policy attributes. If we want to send both, we have to
> send
>     2 BMP messages. If the attributes are unaltered, we still have to
> send
>     2 messages.
>     I'd like to see 2 bits in stead of 1 bit. One for post-policy, and a
> new
>     bit for pre-policy. This will allow us to report the same route only
> once
>     in stead of twice.
>
> 2) It might be nice to have a bit indicating that a post-policy route
>     was denied into the BGP-RIB by policy. This could also be done via a
>     TLV in the new route-monitoring messages. So maybe we shouldn't use a
> bit
>     for this.
>
> 3) This is a personal one. When parsing a BGP update-message in a BMP
>     route-monitoring message, you need to know if the prefix has a
> path-id
>     in front of it or not. The only way to know this is by looking at the
>     OPEN messages in the BMP peer-up messages. This is fine for elaborate
>     bmp-station implementations like OpenBMP. But for state-less software
>     that looks at BMP messages, it is impossible to parse route-mon
> messages.
>     This is a problem for sniffers like WireShark or tcpdump. E.g. the
> peer-up
>     message might have been sent hours or days ago. Also developing
> simple
>     tools for testing or trouble-shooting or development is a pain when
>     add-paths is configured.
>     Therefor I'd like to dedicate one bit in the flags-field to indicate
>     that the NLRI inside the bgp-updates messages have a path-id.
>     We have a similar bit to distinguish between 2-byte and 4-byte ASNs.
> Having
>     a bit to indicate add-paths is kinda the same.
>
>
> My apologies for the long email.
> Thanks for your attention,
>
> henk.
>
>
> ====
> About myself (because this is the first email I'm sending to the IETF in
> 20 years).
> ----
> I work for a router-vendor.
> At the IETF I represent myself, not my employer.
> Lately I've been working on a BMP implementation.
> Router-side, where BGP in a router sends BMP-messages to bmp-stations.
> (For those who recognize my employer's version numbering:
>   our BMP will be released 1-2 months from now, in release 16.0R1).
>
>
> _______________________________________________
> GROW mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/grow
>

---

From: "Guyunan (Yunan Gu, IP Technology Research Dept. NW)" <[email-masked]>
Date: Mon, 23 Apr 2018 12:18:09 +0000

Dear WG,

I agree with Henk in his previous email that a new message type can be defined for carrying the local-rib.

Typically, it’s implementation-efficient to reuse the existing message type for carrying new information, however, as I try to understand the interpretation/mapping of the local-rib information into each existing message type, I find it a little bit difficult to follow. More specifically, the peer definition for the local-rib per-peer head is changed from the “actual peer” to an emulated peer.  Since all peers are emulated, the associated peer up/down notifications and route monitoring messages are all fabricated. I understand that such implementation can be technically workable, but for me it’s a little bit deviated from the “monitoring” purpose of BMP. In fact, the above mentioned concerns can be solved by a straightforward idea of proposing a new message type for carrying the local-rib. Thus, we can save the inconvenience of transforming the BGP speaker–based local-rib into the peer-based format messages.

Back to the new “local-rib message”, as Henk also suggested, the new message can be designed as TLV based. I think that using the TLV based format is more flexible for carrying non-route information in addition to the route information within either local-rib, rib-in or rib-out messages. For example, in order to better understand how the routing policy configured actually worked, an action/policy TLV can be defined and added to each route. The need for carrying the routing policy and actions/decisions taken has also been raised by Ruediger and Thomas in previous emails. I think it will be quite beneficial to the operators to understand how their routing policies actually worked and thus it’s worth taking the need into consideration during the BMP design. In addition, considering the format consistency perspective, except for the pre-policy rib-in, which can be encapsulated using the original BGP Update PDUs and sent to the monitoring station, all the other rib, i.e., post-policy, local-rib and rib-out, need to be first transformed from the rib format into the Update PDU format and then encapsulated with BMP headers, thus we might as well transform the rib into TLVs.

Besides the above comments, I also have one more question regarding the idea of using the new Local-Rib instance peer. Different peers are emulated for different VRF instances and for different address families.  Although it’s mentioned in Section 6.1.1 that different peers are emulated for multiple address families of the same local-rib, there is currently no flag in the per peer header indicating address family (Previously, in RFC7854, the flag V in the per peer header is used to distinguish IPv4 and IPv6). So can the authors explain how to distinguish different address families in both peer up/down notifications for the emulated peers?

Best regards,

Yunan Gu

Huawei Technologies Co. Ltd
Beijing
IP Technology Research Division

From: GROW [mailto:[email-masked]] On Behalf Of Job Snijders
Sent: 2018年4月22日 8:28
To: Henk Smit <[email-masked]>
Cc: [email-masked]
Subject: Re: [GROW] The new BMP drafts and new BMP functionality

Dear WG,

I’d like to encourage the authors and stakeholders of current BMP work to assess the below feedback and get back to the working group.

Kind regards,

Job

On Mon, 26 Mar 2018 at 15:24, Henk Smit <[email-masked]<mailto:[email-masked]>> wrote:

Hello all,

What I propose:
----

    This is a problem for sniffers like WireShark or tcpdump. E.g.. the

henk.

====

_______________________________________________
GROW mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/grow

---

From: "Tim Evens (tievens)" <[email-masked]>
Date: Mon, 23 Apr 2018 16:22:02 +0000

Henk,

On 3/26/18, 6:13 AM, "GROW on behalf of Henk Smit" <[email-masked] on behalf of [email-masked]> wrote:

<tievens> Utilizing a flag for this follows RFC7854 in that the peer being
monitored can convey either Adj-RIB-In or Adj-RIB-Out for Pre-Policy and
Post-Policy, hence flags L and O.  
   
===

<tievens> I believe you missed that loc-rib draft updates RFC7854 by
making flags specific to the peer type.  We updated 7854 to change flags
to be specific by peer type while also not breaking the current
7854 draft/bmp version support of the existing peer types.  Hence,
the existing peer flags are unchanged and therefore do not require
a version change in BMP. Flags by peer type is introduced in
local-rib draft, which specifically introduces a new peer type to 
support this without breaking anything else. 

Below is from draft-ietf-grow-bmp-local-rib-01 where it indicates
flags are specific to the peer type. In other words, 8 bits available
per peer type.   I can see how the below can be overlooked, so we can update
the draft sections 4.1 and 4.2 to clearly indicate that flags are now
specific by peer type.

   8.2.  BMP Peer Flags

      This document defines a new flag (Section 4.2) and proposes that peer
      flags are specific to the peer type:

      o  The F flag indicates that the Loc-RIB is filtered.  This indicates
         that the Loc-RIB does not represent the complete routing table.

===

<tievens> Revision 01 changed this statement to specifically call out why a 
new peer type was introduced.  If this isn't clear, we can further clarify it
more, including adding that that flags are specific to the peer
type, not global. 

"A new peer type is defined for Loc-RIB to distinguish that it
   represents Loc-RIB with or without RD and local instances.
   Section 4.2 [RFC7854] defines a Local Instance Peer type, which is
   for the case of non-RD peers that have an instance identifier."

I agree that local-instance needs to be better clarified in RFC7854.  
We discussed with John why RFC7854 introduced local-instance peer and how it
is a bit misleading.  He mentioned it was added to support local instance/RD
peers (e.g. local vrf) where there isn't a route-distinguisher. Instead
a locally unique RD would be used.  In this local-instance case, it still can represent
a peer.   

BMP adoption by vendors and open-source (FRR, Bird, ...) has been moving at glacial 
rates, including Nokia's lack of support. While you might say you don't like the drafts,
including 7854, and that maybe it's the reason why vendors haven't implemented it... I would 
disagree.  We have been working with both vendors and customers for the past 3 years to 
implement BMP.  The reason why BMP is not implemented is NOT because the drafts
don't work or cannot be implemented as defined, it's because vendors 
(including BIRD and FRR) didn't have the customer demand to prioritize it.  We have
been advocating with both vendors and customers to adopt BMP as the prefer method
for BGP monitoring.  Customers are now finally requesting it, which is likely why
you now have priority to implement it.

Changing the version, when it doesn't really need to changed, adds unnecessary
backwards compatibility complexity with existing and new implementations.  This
effects both sender and receiver implementations.  While I can update SNAS/OpenBMP
to handle the complexity with this, other large cloud/service providers have already
implemented their own non-public BMP receivers that would also have to be updated.  

Change of RFC7854 local-instance usage borders on, if not mandates, a version
changed.  Instead, I’m for a bis update, which we have been talking about with
John, to clarify the use of local-instance peer, that or deprecate it.  

It should be NOTED that there are many items we have talked about updating in
the RFC7854bis.  I would prefer a separate thread to talk about the bis related
items.  

=== 

The other comments are related to RFC7854 and suggest significant changes
requiring a re-write of BMP. Your comments have merit and we have
been talking about this regarding a BIS update instead of a version change. 

I would like John and Jeff's comments on this, but if a version change
is the direction we would like to go, then I would like to discuss all the
changes we have been talking about for the bis updates, not just the 
few you mentioned below.  

--Tim

---

From: "Tim Evens (tievens)" <[email-masked]>
Date: Mon, 23 Apr 2018 22:11:05 +0000

Hi Yunan Gu,

On 4/23/18, 5:18 AM, "GROW on behalf of Guyunan (Yunan Gu, IP Technology Research Dept. NW)" <mailto:[email-masked] on behalf of mailto:[email-masked]> wrote:

<tievens> I believe what you wrote is what we are proposing.  See below response
to emulated peer.  Implementations vary and how "one" generates the BMP common + peer
header as it is not dictated in RFC7854.  For example, as per the current RFC7854
draft, for efficiency the peer header only needs to be initialized once (cached) 
at PEER UP. This includes open messages, peer type, peer flags, RD, info
TLV's, etc. In this case, setting the message type or flag is no more efficient.   

<tievens> Emulated peer is a pseudo logical peer that "represents" the local-rib.
Implementations that do this today, leverage "existing code," requiring little effort
to implement local-rib.  They do this by generating a local-rib peer that is like any
other peer with the exception of no state/session or peer type (iBGP vs eBGP)
restriction.  The update-group (of 1) adj-rib-out for this peer represents the
local-rib. The draft keeps it open to allow the sender to apply filters on this
peer, such as to exclude any IBGP learnt prefixes. In this case with filters,
a single flag indicates the conveyed local-rib does not represent the entire set.
All the drafts (7854, local-rib, adj-rib-out) do not attempt to address configuration
or policy conveyance, other than simple hints via INFO TLV's. IMHO, attempting to
convey and consequently standardize on policy configuration goes into the weeds.
This includes even a simple attempt to convey just the policy names, as names,
order, actions, etc... are not standard over all platforms/vendors. 
While I completely agree operators need to know which policy results in which
change/filter/etc... the conveyance of such does not have to be via BMP.  I would
rather use BMP to monitor and measure the results and use a global unique ID for
correlation of peers to policy/configuration. This would enable monitoring
to be correlated correctly to YANG/Openconfig configurations and policies.  IMO,
it's not just the egress NLRI/attribute policies, it's all of the configuration
that makes up the policy, such as setting next-hop self, add-paths, ...  We 
need more than peer specific policy config, we need the rest of the configuration
(e.g. VRF imports/exports). I do not believe we should overload BMP with 
configuration and policies conveyance.  BMP is to monitor BGP
(NLRI's and attributes) not convey policies or configurations. 

===

<tievens> First, changing the peer-header to TLV's is a fundamental change to
RFC7854 peer header, forcing a version change.  Second, this IMHO is a total
scope creep for BMP as BMP is the wrong place to be conveying policies which
are configurations and are vendor/platform specific.   Adding policy/configuration
to BMP opens the door to a whole new level of complexity with the BMP receiver
now having to handle configurations/policies.  This is normally handled by a 
SDN controller, NFV, or similar product that focuses on configuration and
policy management.  I do not believe those folks will appreciate that policy
will now be moved from, or in parallel with, MDT/OpenConfig.  

<tievens> IMHO, what we need to do is not rewrite what's working to convey
configuration. Instead we should add what's needed to correlate between data
methods and feeds. This enables BMP to support existing and new applications
that specialize in the configuration/policy domain (e.g. SDN controllers,
NFV, etc.).  For example, BGP-LS includes link-id's which can be used to
correlate LS LINK NLRI's to logical/physical links/configurations.  This is great
and works well to support micro services/applications.  The problem, and IMO oversight,
with BGP-LS is that it's not required per the draft to include link-id's, or even
implement them.  This makes it horribly error prone to correlate links to
configuration and other forms of telemetry (e.g. correlate LAG to member links). Yes,
we need to correlate more than just policy configurations, we need to correlate
telemetry interface, flow export data, etc... to BGP monitoring as well.  If we follow
suit with adding non-route data, where is the line in the stand? 

<tievens> Currently, 7854 only gives us BGP ID, RD, and Address to correlate with.
While this does work in most cases, these can be overloaded/duplicated/invalid. 
This can result in correlation errors leading to mistrust when correlating to
datasets such as telemetry and configuration.  I'm leaning more towards adding a
correlation unique ID (maybe associated array if needed) that enables us to correctly
correlate between the related datasets (telemetry, configuration, SDN controllers).

===

<tievens> That's an incorrect understanding of the V flag. The V flag,
as defined in RFC7854, DOES NOT represent the afi/safi's conveyed.  The
V flag only represents the transport session address family. This is why
it is only IPv4 or IPv6 as those are the only transport address families
currently used. It directly is used when decoding the address within the
BMP peer header.  In other words, the V flag only indicates if the peer
header address is IPv4 or IPv6, not the BGP update message families.  

Thanks,
Tim

---

From: Henk Smit <[email-masked]>
Date: Mon, 30 Apr 2018 17:01:15 +0200

Hello Tim,

My apologies for the delayed response.

> While you might say you don't like the drafts, including 7854, and
> that maybe it's the reason why vendors haven't implemented it...

I do like RFC7854. And I do like the ideas behind the two new drafts.
I only have a problem with the syntax. It's not extensible.

I think we should alter the format of route-monitoring messages in BMP.
I'd like to see a syntax that would allow us in the future to add any
information to BMP that becomes required.
And I think we can do it with minimal changes to the current protocol.
And with minimal changes to implementations.

> I would disagree.  We have been working with both vendors and customers
> for the past 3 years to implement BMP.  The reason why BMP is not
> implemented is NOT because the drafts don't work or cannot be 
> implemented
> as defined, it's because vendors  (including BIRD and FRR) didn't have
> the customer demand to prioritize it.

I'm not saying the drafts don't work.
I'm trying to look ahead. And I am convinced that route-monitoring
messages should be TLV based. And if we agree that we have to make that
change some day, I think we should change it asap.

> Changing the version, when it doesn't really need to changed, adds
> unnecessary backwards compatibility complexity with existing and new
> implementations.

Then we shouldn't bump the version.

I think it's a small effort to add a knob in routers to advertise the
current BMP format, or use a new format with three new TLV-based
route-monitoring messages. After all, all other messages stay the same.
And the route-monitoring messages would just get a different 
message-type
and a TLV-header inserted. That is all.

> This effects both sender and receiver implementations.  While I can
> update SNAS/OpenBMP to handle the complexity with this, other large
> cloud/service providers have already implemented their own non-public
> BMP receivers that would also have to be updated.

Agreed.
But again, it wouldn't be such a big change. And routers can easily be
programmed to send either the new or the old syntax. It's a relatively
small change.

> It should be NOTED that there are many items we have talked about 
> updating
> in the RFC7854bis. I would prefer a separate thread to talk about the
> bis related items.

I have not seen that discussion, sorry. Are those discussions archived
in the grow-mailing-list archive ?
I hope I can participate in future discussions.

> changes we have been talking about for the bis updates, not just the

My point was not to bump BMP version's number.

Let me re-iterate my points.
But this time in the reverse order of reasoning.

1) Route-monitoring messages are not extensible.
    If we want to make BMP truly extensible, we should make 
route-monitoring
    messages also TLV-based.
    Look at BGP (attributes) and IS-IS (TLVs and sub-TLVs).
    Those protocols have been unchanged since the early nineties.
    Only new attributes, TLVs and sub-TLVs have been added.
    I'd like BMP to have a similar long life-time.

2) I don't like it if things like peer-flags in the peer-header have
    different meaning based on context. Or message-content not
    depending on msg-type, but on peer-type. It'll make things less 
clear.
    It'll prevent us from doing sensible things in the future.
    I am also afraid that we'll run out of bits in the peer-flags field.

3) Therefor I'd like to split the peer-flags.
    Some flags are indeed about the peer (e.g. ipv4 vs ipv6 
peer-address).
    Others are not about the peer, but about the route (pre/post policy, 
4b-asn).
    Let's use the peer-flags only for information about the peer.
    And when we introduce TLVs for route-monitoring messages, let's make
    a TLV for flags in route-mon messages regarding the routes 
themselves.
    The benefit here is that such a TLV can hold a huge amount of flags.
    (256 bytes * 8 bits = potentially 2048 flags in one TLV).

4) We have to make sure receivers know what to expect. The major thing 
is
    whether route-monitoring messages use TLVs or not. And it has to be
    done in a backward compatible way. We can do that by introducing 1 
new
    msg-type, or 3 new msg-types (Adj-Rib-in, Adj-Rib-out and Loc-Rib), 
or
    by bumping the BMP version number.
    I have no real preference here. I think splitting route-mon msgs into
    three would be the cleanest solution. In that case, we don't need to
    bump the version number.

Summary:
1) split route-monitoring message into 3 msgs: Rib-Adj-in, Rib-Adj-out, 
Loc-rib.
2) make route-mon message TLV-based
3) the only 2 TLVs we need now are: a) BGP update message itself. b) 
flags.

That's all. I don't think this would require much effort to change 
router or
collector implementations.

henk.