Subject: Re: BFD over VXLAN: Trapping BFD Control packet at VTEP

From: Santosh P K <[email-masked]>
Date: Thu, 12 Sep 2019 22:40:49 +0530

Greg,
   Sorry for delay. I will come up with appropriate text for few of the
points listed.

Thanks
Santosh P K

On Fri, Aug 16, 2019 at 4:59 AM Greg Mirsky <[email-masked]> wrote:

> Hi Santosh,
> thank you for your comments. Please find my notes in-lined and tagged
> GIM>>.
>
> Regards,
> Greg
>
> On Tue, Aug 13, 2019 at 10:24 PM Santosh P K <[email-masked]>
> wrote:
>
>> Greg,
>>    Thanks for updated version of document. Here are few comments on new
>> draft.
>>
>> Section 4:
>> Destination MAC: This MUST NOT be of one of tenant's MAC
>>          addresses.  The MAC address MAY be configured, or it MAY be
>>          learned via a control plane protocol.  The details of how the
>>          MAC address is obtained are outside the scope of this document.
>>
>> I think we may need to give background on why we are saying MAC address
>> MUST not be one of tenant's MAC address. Like in this thread we have
>> discussed one of the tenant could have borrowed the same VTEP mac address
>> and we if we have to use BFD then we need to avoid that conflict to ensure
>> BFD packets get observed in the VTEP itself. Should we add a section before
>> 4 to set that context so that above text makes more sense in that context?
>>
> GIM>> Certainly. Please share the text you'd like to add.
>
>>
>>
>>    IP header:
>>          Destination IP: IP address MUST NOT be of one of tenant's IP
>>          addresses.  IP address MAY be selected from the range 127/8 for
>>          IPv4, for IPv6 - from the range 0:0:0:0:0:FFFF:7F00:0/104.
>>
>>          TTL: MUST be set to 1 to ensure that the BFD packet is not
>>          routed within the L3 underlay network.
>>
>>
>> I think we have added some text to address Sridhar comments on why TTL
>> MUST be 1 and dest IP address MUST be 127/8 range. I see that text is
>> missing now.
>>
> GIM>> My apologies that I've missed to include the text from another
> discussion thread. I believe the following would be complete:
>           TTL or Hop Limit: MUST be set to 1 to ensure that the BFD
>          packet is not routed within the Layer 3 underlay network.  This
>          addresses the scenario when the inner IP destination address is
>          of VXLAN gateway and there is a router in underlay which
>          removes the VXLAN header, then it is possible to route the
>          packet as VXLAN  gateway address is routable address.
>>
>>
>> Section 5.1:
>> For such packets, the BFD session MUST be identified
>>    using the following three-tuples of fields of the inner header: the
>>    source IP, the destination IP, and the source UDP port number present
>>    in the IP header carried by the payload of the packet in VXLAN
>>    encapsulation.  If BFD packet is received with non-zero Your
>> Discriminator, then BFD session MUST be demultiplexed only with Your
>>    Discriminator as the key.
>>
>> Just with 3 tuple we will not be able to demux packet. We need to
>> consider VNI as well if we have multiple BFD session between same pair of
>> VTEP.
>>
> GIM>> This is one of comments from Carlos we need to address. Your comment
> have helped me to form the question:
>
> What is the goal running multiple BFD sessions between the pair of VTEPs?
>
> If the goal is to monitor per VNI, then the following text should describe
> the demultiplexing of the initial BFD Control packet:
>    Demultiplexing of IP BFD packet has been defined in Section 3 of
>    [RFC5881].  Since multiple BFD sessions may be running between two
>    VTEPs, there needs to be a mechanism for demultiplexing received BFD
>    packets to the proper session.  For demultiplexing packets with Your
>    Discriminator equal to 0, a BFD session MUST be identified using the
>    logical link over which the BFD Control packet is received.  In the
>    case of VXLAN, the VNI number identifies that logical link.  If BFD
>    packet is received with non-zero Your Discriminator, then BFD session
>    MUST be demultiplexed only with Your Discriminator as the key.
> Would there be need to run multiple BFD sessions with the same VNI number?
>
>
>>
>> Thanks
>> Santosh P K
>>
>>
>> On Fri, Aug 9, 2019 at 4:27 AM Greg Mirsky <[email-masked]> wrote:
>>
>>> Dinesh, thank you for your help, much appreciated.
>>>
>>> Hi Joel and Sridhar,
>>> could you please check if the updated text on the inner Ethernet frame
>>> addressed your concern.
>>>
>>> On Wed, Aug 7, 2019 at 2:25 PM Dinesh Dutt <[email-masked]> wrote:
>>>
>>>> Looks god to me Greg. Thank you for your hard work in this,
>>>>
>>>> Dinesh
>>>>
>>>> On Wed, Aug 7, 2019 at 9:25 AM Greg Mirsky <[email-masked]>
>>>> wrote:
>>>>
>>>>> Hi Dinesh, Joel, Sridhar, et al.,
>>>>> much appreciate the help you've given me sharing your expertise. I
>>>>> hope that the updates you will find in the attached diff and the working
>>>>> copy of the draft be closer to the acceptable solution for VTEP-VTEP BFD.
>>>>> Please note, that I'll shortly start a new discussion thread to address one
>>>>> of Carlos's questions on the ambiguity of the text on multiple concurrent
>>>>> sessions between the same pair of VTEPs.
>>>>> Please review the changes to Sections 4 and 6 and share your feedback,
>>>>> suggestions, and questions.
>>>>>
>>>>> Regards,
>>>>> Greg
>>>>>
>>>>> On Mon, Aug 5, 2019 at 6:03 PM Dinesh Dutt <[email-masked]> wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> On Mon, Aug 5, 2019 at 5:56 PM Greg Mirsky <[email-masked]>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Dinesh,
>>>>>>> thank you for your expedient detailed response.
>>>>>>> I believe that the ability to run BFD session up to a tenant
>>>>>>> (VTEP-VTEP-tenant or tenant-tenant) was never in jeopardy from this
>>>>>>> specification.
>>>>>>> I'm trying to provide precise specification on what can be used ad
>>>>>>> the destination MAC and IP addresses in the inner frame/packet as I believe
>>>>>>> that likely will help to avoid interoperability issues.
>>>>>>> I'm interested to learn some more about the "martian checking"
>>>>>>> function. As you know, martian addresses have been used as destination IP
>>>>>>> address in LSP Ping and BFD over MPLS LSP and PW. I haven't heard that any
>>>>>>> silicon feature caused problems for operators using these tools.
>>>>>>>
>>>>>>
>>>>>> Interesting. I didn't know this aspect of use with MPLS ping. Did
>>>>>> those packets ever go through a firewall though? In any case, maybe suggest
>>>>>> the use of those addresses with a statement that this is how LSP does it,
>>>>>> but that other MAC/IP pairs are possible as long as the conditions of the
>>>>>> endpoint owning the MAC/IP was honored.
>>>>>>
>>>>>> Dinesh
>>>>>>
>>>>>>>
>>>>>>> Regards,
>>>>>>> Greg
>>>>>>>
>>>>>>> On Mon, Aug 5, 2019 at 3:59 PM Dinesh Dutt <[email-masked]> wrote:
>>>>>>>
>>>>>>>> Hi Greg,
>>>>>>>>
>>>>>>>> That we agree on the problem definition is the first step forward.
>>>>>>>> Your original document had my cases covered and so I was surprised by the
>>>>>>>> track this thread took. It doesn't matter, we're back on track.
>>>>>>>>
>>>>>>>> My recommendation is to not worry about specifying the precise
>>>>>>>> MAC/IP address used in the inner header. The addresses chosen MUST ensure
>>>>>>>> that the packet is trapped to the control plane of the VTEP and not escape
>>>>>>>> to the tenant if the BFD is to the VTEP. Any solution MUST also not
>>>>>>>> preclude the use of the BFD by tenant systems for that VNI. There are many
>>>>>>>> ways an implementer can choose to implement this. For example, the inner
>>>>>>>> MAC address is whatever the VTEP implementer would return if ARP'd for the
>>>>>>>> IP address used in the inner header in the given VNI. The implementer can
>>>>>>>> pick a fixed MAC address, one that they own etc. Multiple BFD sessions can
>>>>>>>> be run for testing path connectivity on more than one VNIs. Limits should
>>>>>>>> be in place to avoid overwhelming the receiver with BFD messages (you had
>>>>>>>> words about this in your currently published draft).  If the VNI is
>>>>>>>> irrelevant in the test i.e. only the VXLAN pipe at the VTEP is being
>>>>>>>> tested. the user can use any VNI active on the VTEP on which the VTEP owns
>>>>>>>> an IP address.
>>>>>>>>
>>>>>>>> I'm concerned about the use of 127/8 address only because of
>>>>>>>> firewalls or implementations that drop packets with these addresses as
>>>>>>>> either the source or destination. For example, on many merchant silicon, I
>>>>>>>> don't believe you can turn off martian checking and drops *only* for
>>>>>>>> VXLAN-encapsulated BFD packets. I don't know what the Linux kernel does
>>>>>>>> today on such packets, for example (or Hyper-V). I'd like a solution that
>>>>>>>> doesn't demand additional or new chip functionality or require additional
>>>>>>>> middle-box hole punch.
>>>>>>>>
>>>>>>>> Why do you feel you MUST to specify the MAC/IP address on the inner
>>>>>>>> packet? What am I missing here?
>>>>>>>>
>>>>>>>> Dinesh
>>>>>>>>
>>>>>>>> On Mon, Aug 5, 2019 at 3:04 PM Greg Mirsky <[email-masked]>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi Dinesh,
>>>>>>>>> what do you see as the way forward? I agree, that the proposed
>>>>>>>>> text doesn't work for multi-VNI concurrent monitoring because these VNIs
>>>>>>>>> are tenant's VNIs. And in that case, we need to specify another mechanism
>>>>>>>>> to trap the BFD Control packet at VTEP. It seems that VTEP's Ethernet
>>>>>>>>> address must be used as the destination MAC address in the inner Ethernet
>>>>>>>>> frame. The destination IP address may be either VTEP's address of martian
>>>>>>>>> (I do prefer martian). Let me give it  try:
>>>>>>>>> NEW TEXT:
>>>>>>>>>
>>>>>>>>> To monitor continuity of the path between two VTEPs, an operator
>>>>>>>>> MUST select a VNI number to be used as Management VNI. Management VNI
>>>>>>>>> number MUST NOT be one of the tenant's VNIs to prevent sending VXLAN
>>>>>>>>> packets received on Management VNI to a tenant. VNI number 1 is RECOMMENDED
>>>>>>>>> as the default for Management VNI. [Ed.note: What we set the Destination
>>>>>>>>> MAC to? Can it be invalid MAC that MUST be ignored on receipt?]
>>>>>>>>>
>>>>>>>>> If an implementation supports concurrent monitoring of multiple
>>>>>>>>> VNIs, then the value of VNI number MAY be one of tenant's VNIs. The
>>>>>>>>> destination MAC address in the inner Ethernet frame encapsulating BFD
>>>>>>>>> Control packet MUST be MAC associated with the remote VTEP.
>>>>>>>>> The destination IP address of the inner IP packet MUST be selected
>>>>>>>>> from the range 127/8 for IPv4, and for IPv6 from the range
>>>>>>>>> 0:0:0:0:0:FFFF:7F00:0/104. The TTL value in the inner IP header MUST be set
>>>>>>>>> to 1.
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Greg
>>>>>>>>>
>>>>>>>>> On Sun, Aug 4, 2019 at 9:07 AM Dinesh Dutt <[email-masked]>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Greg,
>>>>>>>>>>
>>>>>>>>>> Thanks for your clarifications. I agree with your sentiment on
>>>>>>>>>> why you're running BFD over VXLAN between VTEPs. I wasn't arguing against
>>>>>>>>>> it at all. All I was saying was pointing to the limitations of the use of
>>>>>>>>>> management VNI. I spoke to some operators who're running EVPN and mentioned
>>>>>>>>>> the discussion on this thread. They concur that they're using specific VNIs
>>>>>>>>>> to test connectivity over that VNI between VTEPs to ensure misconfiguration
>>>>>>>>>> doesn't lead to blackholes. My statements are based in real world operator
>>>>>>>>>> experience. And I was providing language that ensured packets didn't leak
>>>>>>>>>> across to tenants when they were destined to VTEPs.
>>>>>>>>>>
>>>>>>>>>> Dinesh
>>>>>>>>>>
>>>>>>>>>> On Sat, Aug 3, 2019 at 10:34 AM Greg Mirsky <
>>>>>>>>>> [email-masked]> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Dinesh,
>>>>>>>>>>> many thanks for your detailed updates on how some
>>>>>>>>>>> implementations process VXLAN header and the inner Ethernet frame. These
>>>>>>>>>>> are very helpful in achieving the workable solution for the problem at hand.
>>>>>>>>>>> You've noted that a path between VTEPs may be monitored in the
>>>>>>>>>>> underlay network by merely establishing a BFD session. That is true, but by
>>>>>>>>>>> using BFD with VXLAN encapsulation between the pair of VTEPs we are
>>>>>>>>>>> extending the OAM domain by including, to some extent, VXLAN forwarding
>>>>>>>>>>> engine. Abstract in RFC 5880 defines the goal and the domain in which BFD
>>>>>>>>>>> protocol can detect a fault as:
>>>>>>>>>>>    This document describes a protocol intended to detect faults
>>>>>>>>>>> in the
>>>>>>>>>>>    bidirectional path between two forwarding engines, including
>>>>>>>>>>>    interfaces, data link(s), and to the extent possible the
>>>>>>>>>>> forwarding
>>>>>>>>>>>    engines themselves, with potentially very low latency.
>>>>>>>>>>> Thus, BFD in the underlay will exercise a part of IP forwarding
>>>>>>>>>>> engine while BFD with VXLAN encapsulation, ran between the same pair of
>>>>>>>>>>> VTEPs, extends the OAM domain. At the same time, defining BFD between
>>>>>>>>>>> tenant systems in outside the goal of this specification. But VXLAN BFD
>>>>>>>>>>> session between VTEPs may be useful in monitoring e2e path between tenants,
>>>>>>>>>>> as described in the update to -07:
>>>>>>>>>>>    At the same time, a service layer BFD session may be used
>>>>>>>>>>> between the
>>>>>>>>>>>    tenants of VTEPs IP1 and IP2 to provide end-to-end fault
>>>>>>>>>>> management.
>>>>>>>>>>>    In such case, for VTEPs BFD control packets of that session
>>>>>>>>>>> are
>>>>>>>>>>>    indistinguishable from data packets.  If end-to-end defect
>>>>>>>>>>> detection
>>>>>>>>>>>    is realized as the set of concatenated OAM domains, e.g.,
>>>>>>>>>>> VM1-1 - IP1
>>>>>>>>>>>    -- IP2 - VM2-1, then the BFD session over VXLAN between VTEPs
>>>>>>>>>>> SHOULD
>>>>>>>>>>>    follow the procedures described in Section 6.8.17 [RFC5880].
>>>>>>>>>>> I've attached the current working version of the draft.
>>>>>>>>>>>
>>>>>>>>>>> Regards,
>>>>>>>>>>> Greg
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Aug 2, 2019 at 5:43 PM Dinesh Dutt <[email-masked]>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> What I mean is "How do you infer that it excludes the case I'm
>>>>>>>>>>>> talking about?".
>>>>>>>>>>>>
>>>>>>>>>>>> Dinesh
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Aug 2, 2019 at 5:41 PM Dinesh Dutt <[email-masked]>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> The abstract reads this: "
>>>>>>>>>>>>>
>>>>>>>>>>>>> This document describes the use of the Bidirectional Forwarding
>>>>>>>>>>>>>    Detection (BFD) protocol in point-to-point Virtual eXtensible Local
>>>>>>>>>>>>>    Area Network (VXLAN) tunnels forming up an overlay network."
>>>>>>>>>>>>>
>>>>>>>>>>>>> How do you infer what you said?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Dinesh
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Fri, Aug 2, 2019 at 5:38 PM Joel M. Halpern <
>>>>>>>>>>>>> [email-masked]> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> I am going by what the draft says its purpose is.  If you
>>>>>>>>>>>>>> (Dinesh) want
>>>>>>>>>>>>>> the draft to fulfill a different purpose, then either ask the
>>>>>>>>>>>>>> chairs to
>>>>>>>>>>>>>> take this draft back to the WG, or write a separate draft.
>>>>>>>>>>>>>> As currently written, the behavior Greg proposed meets the
>>>>>>>>>>>>>> needs, and
>>>>>>>>>>>>>> does so in a way that is consistent with VxLAN.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Yours,
>>>>>>>>>>>>>> Joel
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 8/2/2019 8:30 PM, Dinesh Dutt wrote:
>>>>>>>>>>>>>> > What is the stated purpose of this BFD session? The VTEP
>>>>>>>>>>>>>> reachability is
>>>>>>>>>>>>>> > determined by the underlay, I don't need VXLAN-encaped
>>>>>>>>>>>>>> packet for that.
>>>>>>>>>>>>>> > Do we agree?
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> > If I want to test the VXLAN encap/decap functionality
>>>>>>>>>>>>>> alone, picking any
>>>>>>>>>>>>>> > single VNI maybe fine. But is this all any network operator
>>>>>>>>>>>>>> wants? Why?
>>>>>>>>>>>>>> > In what situations has this been a problem? I suspect
>>>>>>>>>>>>>> operators also
>>>>>>>>>>>>>> > want to verify path continuity over a specific VNI. If you
>>>>>>>>>>>>>> say this is
>>>>>>>>>>>>>> > not defined by the document, I disagree because the current
>>>>>>>>>>>>>> version
>>>>>>>>>>>>>> > talks about controlling the number of BFD sessions between
>>>>>>>>>>>>>> the VTEPs
>>>>>>>>>>>>>> > (see section 3). More importantly, this is a real problem
>>>>>>>>>>>>>> that operators
>>>>>>>>>>>>>> > like to verify.
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> > Dinesh
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> > On Fri, Aug 2, 2019 at 5:08 PM Joel M. Halpern <
>>>>>>>>>>>>>> [email-masked]
>>>>>>>>>>>>>> > <mailto:[email-masked]>> wrote:
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> >     What is special about the management VNI is precisely
>>>>>>>>>>>>>> that it is NOT a
>>>>>>>>>>>>>> >     tenant VNI.  The VxLAN administration does know how it
>>>>>>>>>>>>>> allocates VNI to
>>>>>>>>>>>>>> >     tenants, and which VNI it has allocated.  In contrast,
>>>>>>>>>>>>>> it does not know
>>>>>>>>>>>>>> >     which IP addresses or MAC adddresses teh tenant is
>>>>>>>>>>>>>> using or may plan
>>>>>>>>>>>>>> >     to use.
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> >     Yours,
>>>>>>>>>>>>>> >     Joel
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> >     On 8/2/2019 6:41 PM, Dinesh Dutt wrote:
>>>>>>>>>>>>>> >      > The assumption of an IP address within any VNI is
>>>>>>>>>>>>>> suspect that way.
>>>>>>>>>>>>>> >      > What's special about a single VNI, the management
>>>>>>>>>>>>>> VNI? The VTEP IP
>>>>>>>>>>>>>> >      > address does not belong in reality in any VNI.
>>>>>>>>>>>>>> >      >
>>>>>>>>>>>>>> >      > Dinesh
>>>>>>>>>>>>>> >      >
>>>>>>>>>>>>>> >      > On Fri, Aug 2, 2019 at 3:17 PM Joel M. Halpern
>>>>>>>>>>>>>> >     <[email-masked] <mailto:[email-masked]>
>>>>>>>>>>>>>> >      > <mailto:[email-masked] <mailto:
>>>>>>>>>>>>>> [email-masked]>>> wrote:
>>>>>>>>>>>>>> >      >
>>>>>>>>>>>>>> >      >     Your response seems to miss two points:
>>>>>>>>>>>>>> >      >
>>>>>>>>>>>>>> >      >     First, the problem you describe is not what the
>>>>>>>>>>>>>> document says
>>>>>>>>>>>>>> >     it is
>>>>>>>>>>>>>> >      >     solving.  To the degree it discusses it at all,
>>>>>>>>>>>>>> the document
>>>>>>>>>>>>>> >     says "
>>>>>>>>>>>>>> >      >       In
>>>>>>>>>>>>>> >      >     most cases, a single BFD session is sufficient
>>>>>>>>>>>>>> for the given
>>>>>>>>>>>>>> >     VTEP to
>>>>>>>>>>>>>> >      >     monitor the reachability of a remote VTEP,
>>>>>>>>>>>>>> regardless of the
>>>>>>>>>>>>>> >     number of
>>>>>>>>>>>>>> >      >     VNIs in common. "
>>>>>>>>>>>>>> >      >
>>>>>>>>>>>>>> >      >     Second, you assume the existence of an IP
>>>>>>>>>>>>>> address for a VTEP
>>>>>>>>>>>>>> >     within a
>>>>>>>>>>>>>> >      >     VNI.  As with the MAC address, the VTEP does not
>>>>>>>>>>>>>> have an IP
>>>>>>>>>>>>>> >     address
>>>>>>>>>>>>>> >      >     within the VNI.  Some implementations may have
>>>>>>>>>>>>>> created such a
>>>>>>>>>>>>>> >     thing,
>>>>>>>>>>>>>> >      >     but
>>>>>>>>>>>>>> >      >     the general construct, as defined to date, does
>>>>>>>>>>>>>> not support such.
>>>>>>>>>>>>>> >      >
>>>>>>>>>>>>>> >      >     In short, you are requiring a behavior that
>>>>>>>>>>>>>> violates the
>>>>>>>>>>>>>> >     architectural
>>>>>>>>>>>>>> >      >     structure of overlay / underlay separation, and
>>>>>>>>>>>>>> common
>>>>>>>>>>>>>> >     usage.  And you
>>>>>>>>>>>>>> >      >     are doing so to support a use case that the
>>>>>>>>>>>>>> working group has not
>>>>>>>>>>>>>> >      >     indicated in the document as important.
>>>>>>>>>>>>>> >      >
>>>>>>>>>>>>>> >      >     Yours,
>>>>>>>>>>>>>> >      >     Joel
>>>>>>>>>>>>>> >      >
>>>>>>>>>>>>>> >      >     On 8/2/2019 5:01 PM, Dinesh Dutt wrote:
>>>>>>>>>>>>>> >      >      > Joel,
>>>>>>>>>>>>>> >      >      >
>>>>>>>>>>>>>> >      >      > You understood correctly.
>>>>>>>>>>>>>> >      >      >
>>>>>>>>>>>>>> >      >      > The VNIs may not share fate due to
>>>>>>>>>>>>>> misconfiguration. And I
>>>>>>>>>>>>>> >     strongly
>>>>>>>>>>>>>> >      >      > suspect someone will want to use BFD for that
>>>>>>>>>>>>>> because its
>>>>>>>>>>>>>> >     about
>>>>>>>>>>>>>> >      >     checking
>>>>>>>>>>>>>> >      >      > path continuity as stated by the draft. As
>>>>>>>>>>>>>> long as there's a
>>>>>>>>>>>>>> >      >     valid IP
>>>>>>>>>>>>>> >      >      > (because it's BFD) owned by the VTEP in that
>>>>>>>>>>>>>> VNI, you can
>>>>>>>>>>>>>> >     use BFD in
>>>>>>>>>>>>>> >      >      > that VNI. Thats all that you need to
>>>>>>>>>>>>>> dictate.  That IP address
>>>>>>>>>>>>>> >      >     has a MAC
>>>>>>>>>>>>>> >      >      > address and you can use that on the inner
>>>>>>>>>>>>>> frame. That is
>>>>>>>>>>>>>> >     all normal
>>>>>>>>>>>>>> >      >      > VXLAN processing. The outer IP is always that
>>>>>>>>>>>>>> of the VTEP.
>>>>>>>>>>>>>> >      >      >
>>>>>>>>>>>>>> >      >      > Dinesh
>>>>>>>>>>>>>> >      >      >
>>>>>>>>>>>>>> >      >      > On Fri, Aug 2, 2019 at 11:03 AM Joel M.
>>>>>>>>>>>>>> Halpern
>>>>>>>>>>>>>> >      >     <[email-masked] <mailto:[email-masked]
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> >     <mailto:[email-masked] <mailto:[email-masked]
>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>> >      >      > <mailto:[email-masked] <mailto:
>>>>>>>>>>>>>> [email-masked]>
>>>>>>>>>>>>>> >     <mailto:[email-masked] <mailto:[email-masked]>>>>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>> >      >      >
>>>>>>>>>>>>>> >      >      >     If I am reading your various emails
>>>>>>>>>>>>>> correctly Dinesh
>>>>>>>>>>>>>> >     (and I
>>>>>>>>>>>>>> >      >     may have
>>>>>>>>>>>>>> >      >      >     missed something) you are trying to use
>>>>>>>>>>>>>> the MAC address
>>>>>>>>>>>>>> >      >     because you
>>>>>>>>>>>>>> >      >      >     want
>>>>>>>>>>>>>> >      >      >     to be able to send these BFD packets over
>>>>>>>>>>>>>> arbitrary VNI to
>>>>>>>>>>>>>> >      >     monitor the
>>>>>>>>>>>>>> >      >      >     VNI.  That is not a requirement
>>>>>>>>>>>>>> identified in the
>>>>>>>>>>>>>> >     document.
>>>>>>>>>>>>>> >      >     It is not
>>>>>>>>>>>>>> >      >      >     even a problem I understand, since all
>>>>>>>>>>>>>> the VNI between an
>>>>>>>>>>>>>> >      >     ingress and
>>>>>>>>>>>>>> >      >      >     egress VTEP share fate.
>>>>>>>>>>>>>> >      >      >
>>>>>>>>>>>>>> >      >      >     Yours,
>>>>>>>>>>>>>> >      >      >     Joel
>>>>>>>>>>>>>> >      >      >
>>>>>>>>>>>>>> >      >      >     On 8/2/2019 1:44 PM, Dinesh Dutt wrote:
>>>>>>>>>>>>>> >      >      >      > Thanks for verifying this. On Linux
>>>>>>>>>>>>>> and hardware
>>>>>>>>>>>>>> >     routers
>>>>>>>>>>>>>> >      >     that I'm
>>>>>>>>>>>>>> >      >      >     aware
>>>>>>>>>>>>>> >      >      >      > of (Cisco circa 2012 and Cumulus), the
>>>>>>>>>>>>>> physical MAC
>>>>>>>>>>>>>> >     address is
>>>>>>>>>>>>>> >      >      >     reused
>>>>>>>>>>>>>> >      >      >      > across the VNIs on the VTEP. Did you
>>>>>>>>>>>>>> check on a non-VMW
>>>>>>>>>>>>>> >      >     device?
>>>>>>>>>>>>>> >      >      >     This is
>>>>>>>>>>>>>> >      >      >      > more for my own curiosity.
>>>>>>>>>>>>>> >      >      >      >
>>>>>>>>>>>>>> >      >      >      > To address the general case, can we
>>>>>>>>>>>>>> not define a
>>>>>>>>>>>>>> >      >     well-known (or
>>>>>>>>>>>>>> >      >      >     reserve
>>>>>>>>>>>>>> >      >      >      > one) unicast MAC address for use with
>>>>>>>>>>>>>> VTEP? If the MAC
>>>>>>>>>>>>>> >      >     address is
>>>>>>>>>>>>>> >      >      >      > configurable in BFD command, this can
>>>>>>>>>>>>>> be moot.
>>>>>>>>>>>>>> >      >      >      >
>>>>>>>>>>>>>> >      >      >      > Dinesh
>>>>>>>>>>>>>> >      >      >      >
>>>>>>>>>>>>>> >      >      >      > On Fri, Aug 2, 2019 at 10:27 AM
>>>>>>>>>>>>>> Santosh P K
>>>>>>>>>>>>>> >      >      >      > <[email-masked]
>>>>>>>>>>>>>> >     <mailto:[email-masked]>
>>>>>>>>>>>>>> >      >     <mailto:[email-masked]
>>>>>>>>>>>>>> >     <mailto:[email-masked]>>
>>>>>>>>>>>>>> >      >      >     <mailto:[email-masked]
>>>>>>>>>>>>>> >     <mailto:[email-masked]>
>>>>>>>>>>>>>> >      >     <mailto:[email-masked]
>>>>>>>>>>>>>> >     <mailto:[email-masked]>>>
>>>>>>>>>>>>>> >      >      >     <mailto:[email-masked]
>>>>>>>>>>>>>> >     <mailto:[email-masked]>
>>>>>>>>>>>>>> >      >     <mailto:[email-masked]
>>>>>>>>>>>>>> >     <mailto:[email-masked]>>
>>>>>>>>>>>>>> >      >      >     <mailto:[email-masked]
>>>>>>>>>>>>>> >     <mailto:[email-masked]>
>>>>>>>>>>>>>> >      >     <mailto:[email-masked]
>>>>>>>>>>>>>> >     <mailto:[email-masked]>>>>> wrote:
>>>>>>>>>>>>>> >      >      >      >
>>>>>>>>>>>>>> >      >      >      >     I have cross checked point raised
>>>>>>>>>>>>>> about MAC address
>>>>>>>>>>>>>> >      >     usage. It is
>>>>>>>>>>>>>> >      >      >      >     possible that tenant could be
>>>>>>>>>>>>>> using physical MAC
>>>>>>>>>>>>>> >      >     address and
>>>>>>>>>>>>>> >      >      >     when a
>>>>>>>>>>>>>> >      >      >      >     packet comes with valid VNI with a
>>>>>>>>>>>>>> MAC address
>>>>>>>>>>>>>> >     that is
>>>>>>>>>>>>>> >      >     being
>>>>>>>>>>>>>> >      >      >     used by
>>>>>>>>>>>>>> >      >      >      >     tenant then packet will be sent to
>>>>>>>>>>>>>> that tenant.
>>>>>>>>>>>>>> >     This rules
>>>>>>>>>>>>>> >      >      >     out the
>>>>>>>>>>>>>> >      >      >      >     fact that we could use physical
>>>>>>>>>>>>>> MAC address as
>>>>>>>>>>>>>> >     inner
>>>>>>>>>>>>>> >      >     MAC to
>>>>>>>>>>>>>> >      >      >     ensure
>>>>>>>>>>>>>> >      >      >      >     packets get terminated at VTEP
>>>>>>>>>>>>>> itself.
>>>>>>>>>>>>>> >      >      >      >
>>>>>>>>>>>>>> >      >      >      >     Thanks
>>>>>>>>>>>>>> >      >      >      >     Santosh P K
>>>>>>>>>>>>>> >      >      >      >
>>>>>>>>>>>>>> >      >      >      >     On Wed, Jul 31, 2019 at 11:00 AM
>>>>>>>>>>>>>> Santosh P K
>>>>>>>>>>>>>> >      >      >      >     <[email-masked]
>>>>>>>>>>>>>> >     <mailto:[email-masked]>
>>>>>>>>>>>>>> >      >     <mailto:[email-masked]
>>>>>>>>>>>>>> >     <mailto:[email-masked]>>
>>>>>>>>>>>>>> >      >      >     <mailto:[email-masked]
>>>>>>>>>>>>>> >     <mailto:[email-masked]>
>>>>>>>>>>>>>> >      >     <mailto:[email-masked]
>>>>>>>>>>>>>> >     <mailto:[email-masked]>>>
>>>>>>>>>>>>>> >      >      >     <mailto:[email-masked]
>>>>>>>>>>>>>> >     <mailto:[email-masked]>
>>>>>>>>>>>>>> >      >     <mailto:[email-masked]
>>>>>>>>>>>>>> >     <mailto:[email-masked]>>
>>>>>>>>>>>>>> >      >      >     <mailto:[email-masked]
>>>>>>>>>>>>>> >     <mailto:[email-masked]>
>>>>>>>>>>>>>> >      >     <mailto:[email-masked]
>>>>>>>>>>>>>> >     <mailto:[email-masked]>>>>>
>>>>>>>>>>>>>> >      >      >      >     wrote:
>>>>>>>>>>>>>> >      >      >      >
>>>>>>>>>>>>>> >      >      >      >         Joel,
>>>>>>>>>>>>>> >      >      >      >             Thanks for your inputs. I
>>>>>>>>>>>>>> checked
>>>>>>>>>>>>>> >      >     implementation within
>>>>>>>>>>>>>> >      >      >      >         Vmware. Perhaps I should have
>>>>>>>>>>>>>> been more clear
>>>>>>>>>>>>>> >      >     about MAC
>>>>>>>>>>>>>> >      >      >     address
>>>>>>>>>>>>>> >      >      >      >         space while checking
>>>>>>>>>>>>>> internally. I will cross
>>>>>>>>>>>>>> >      >     check again for
>>>>>>>>>>>>>> >      >      >      >         the same and get back on this
>>>>>>>>>>>>>> list.
>>>>>>>>>>>>>> >      >      >      >
>>>>>>>>>>>>>> >      >      >      >         Thanks
>>>>>>>>>>>>>> >      >      >      >         Santosh P K
>>>>>>>>>>>>>> >      >      >      >
>>>>>>>>>>>>>> >      >      >      >         On Wed, Jul 31, 2019 at 10:54
>>>>>>>>>>>>>> AM Joel M.
>>>>>>>>>>>>>> >     Halpern
>>>>>>>>>>>>>> >      >      >      >         <[email-masked]
>>>>>>>>>>>>>> >     <mailto:[email-masked]> <mailto:
>>>>>>>>>>>>>> [email-masked]
>>>>>>>>>>>>>> >     <mailto:[email-masked]>>
>>>>>>>>>>>>>> >      >     <mailto:[email-masked] <mailto:
>>>>>>>>>>>>>> [email-masked]>
>>>>>>>>>>>>>> >     <mailto:[email-masked] <mailto:[email-masked]
>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>> >      >      >     <mailto:[email-masked]
>>>>>>>>>>>>>> >     <mailto:[email-masked]> <mailto:
>>>>>>>>>>>>>> [email-masked]
>>>>>>>>>>>>>> >     <mailto:[email-masked]>>
>>>>>>>>>>>>>> >      >     <mailto:[email-masked] <mailto:
>>>>>>>>>>>>>> [email-masked]>
>>>>>>>>>>>>>> >     <mailto:[email-masked] <mailto:[email-masked]>>>>>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>> >      >      >      >
>>>>>>>>>>>>>> >      >      >      >             Sorry to ask a stupid
>>>>>>>>>>>>>> question.  Whose
>>>>>>>>>>>>>> >      >     implementation?
>>>>>>>>>>>>>> >      >      >      >
>>>>>>>>>>>>>> >      >      >      >             The reason I ask is that
>>>>>>>>>>>>>> as far as I
>>>>>>>>>>>>>> >     can tell,
>>>>>>>>>>>>>> >      >     since the
>>>>>>>>>>>>>> >      >      >      >             tenant does not
>>>>>>>>>>>>>> >      >      >      >             have any control access to
>>>>>>>>>>>>>> the VTEP,
>>>>>>>>>>>>>> >     there is no
>>>>>>>>>>>>>> >      >      >     reason for
>>>>>>>>>>>>>> >      >      >      >             the VTEP to
>>>>>>>>>>>>>> >      >      >      >             have a MAC address in the
>>>>>>>>>>>>>> tenant
>>>>>>>>>>>>>> >     space.  Yes, the
>>>>>>>>>>>>>> >      >      >     device has
>>>>>>>>>>>>>> >      >      >      >             a physical
>>>>>>>>>>>>>> >      >      >      >             MAC address.  But the
>>>>>>>>>>>>>> tenant could well be
>>>>>>>>>>>>>> >      >     using that MAC
>>>>>>>>>>>>>> >      >      >      >             address.  Yes,
>>>>>>>>>>>>>> >      >      >      >             they would be violating
>>>>>>>>>>>>>> the Ethernet spec.
>>>>>>>>>>>>>> >      >     But the whole
>>>>>>>>>>>>>> >      >      >      >             point of
>>>>>>>>>>>>>> >      >      >      >             segregation is not to care
>>>>>>>>>>>>>> about such
>>>>>>>>>>>>>> >     issues.
>>>>>>>>>>>>>> >      >      >      >
>>>>>>>>>>>>>> >      >      >      >             On the other hand, if you
>>>>>>>>>>>>>> tell me that
>>>>>>>>>>>>>> >     the VMWare
>>>>>>>>>>>>>> >      >      >      >             implementation has an
>>>>>>>>>>>>>> >      >      >      >             Ethernet address that is
>>>>>>>>>>>>>> part of the tenant
>>>>>>>>>>>>>> >      >     space, well,
>>>>>>>>>>>>>> >      >      >      >             they made up
>>>>>>>>>>>>>> >      >      >      >             this particular game.
>>>>>>>>>>>>>> >      >      >      >
>>>>>>>>>>>>>> >      >      >      >             Yours,
>>>>>>>>>>>>>> >      >      >      >             Joel
>>>>>>>>>>>>>> >      >      >      >
>>>>>>>>>>>>>> >      >      >      >             On 7/31/2019 1:44 PM,
>>>>>>>>>>>>>> Santosh P K wrote:
>>>>>>>>>>>>>> >      >      >      >              > I have checked with
>>>>>>>>>>>>>> implementation
>>>>>>>>>>>>>> >     in data
>>>>>>>>>>>>>> >      >     path.
>>>>>>>>>>>>>> >      >      >     When we
>>>>>>>>>>>>>> >      >      >      >             receive a
>>>>>>>>>>>>>> >      >      >      >              > packet with valid VNI
>>>>>>>>>>>>>> then lookup
>>>>>>>>>>>>>> >     for MAC will
>>>>>>>>>>>>>> >      >      >     happen and
>>>>>>>>>>>>>> >      >      >      >             it is VTEP own
>>>>>>>>>>>>>> >      >      >      >              > MAC then it will be
>>>>>>>>>>>>>> trapped to control
>>>>>>>>>>>>>> >      >     plane for
>>>>>>>>>>>>>> >      >      >      >             processing. I think we
>>>>>>>>>>>>>> >      >      >      >              > can have following
>>>>>>>>>>>>>> options
>>>>>>>>>>>>>> >      >      >      >              > 1. Optional managment
>>>>>>>>>>>>>> VNI
>>>>>>>>>>>>>> >      >      >      >              > 2. Mandatory inner MAC
>>>>>>>>>>>>>> set to VTEP mac
>>>>>>>>>>>>>> >      >      >      >              > 3. Inner IP TTL set to
>>>>>>>>>>>>>> 1 to avoid
>>>>>>>>>>>>>> >      >     forwarding of packet
>>>>>>>>>>>>>> >      >      >      >             via inner IP
>>>>>>>>>>>>>> >      >      >      >              > address.
>>>>>>>>>>>>>> >      >      >      >              >
>>>>>>>>>>>>>> >      >      >      >              >
>>>>>>>>>>>>>> >      >      >      >              > Thoughts?
>>>>>>>>>>>>>> >      >      >      >              >
>>>>>>>>>>>>>> >      >      >      >              > Thansk
>>>>>>>>>>>>>> >      >      >      >              > Santosh P K
>>>>>>>>>>>>>> >      >      >      >              >
>>>>>>>>>>>>>> >      >      >      >              > On Wed, Jul 31, 2019 at
>>>>>>>>>>>>>> 9:20 AM Greg
>>>>>>>>>>>>>> >     Mirsky
>>>>>>>>>>>>>> >      >      >      >             <[email-masked]
>>>>>>>>>>>>>> >     <mailto:[email-masked]>
>>>>>>>>>>>>>> >      >     <mailto:[email-masked] <mailto:
>>>>>>>>>>>>>> [email-masked]>>
>>>>>>>>>>>>>> >     <mailto:[email-masked] <mailto:
>>>>>>>>>>>>>> [email-masked]>
>>>>>>>>>>>>>> >      >     <mailto:[email-masked] <mailto:
>>>>>>>>>>>>>> [email-masked]>>>
>>>>>>>>>>>>>> >      >      >     <mailto:[email-masked]
>>>>>>>>>>>>>> >     <mailto:[email-masked]> <mailto:
>>>>>>>>>>>>>> [email-masked]
>>>>>>>>>>>>>> >     <mailto:[email-masked]>>
>>>>>>>>>>>>>> >      >     <mailto:[email-masked] <mailto:
>>>>>>>>>>>>>> [email-masked]>
>>>>>>>>>>>>>> >     <mailto:[email-masked] <mailto:
>>>>>>>>>>>>>> [email-masked]>>>>
>>>>>>>>>>>>>> >      >      >      >              > <mailto:
>>>>>>>>>>>>>> [email-masked]
>>>>>>>>>>>>>> >     <mailto:[email-masked]>
>>>>>>>>>>>>>> >      >     <mailto:[email-masked] <mailto:
>>>>>>>>>>>>>> [email-masked]>>
>>>>>>>>>>>>>> >      >      >     <mailto:[email-masked]
>>>>>>>>>>>>>> >     <mailto:[email-masked]> <mailto:
>>>>>>>>>>>>>> [email-masked]
>>>>>>>>>>>>>> >     <mailto:[email-masked]>>>
>>>>>>>>>>>>>> >      >      >      >             <mailto:
>>>>>>>>>>>>>> [email-masked]
>>>>>>>>>>>>>> >     <mailto:[email-masked]>
>>>>>>>>>>>>>> >      >     <mailto:[email-masked] <mailto:
>>>>>>>>>>>>>> [email-masked]>>
>>>>>>>>>>>>>> >      >      >     <mailto:[email-masked]
>>>>>>>>>>>>>> >     <mailto:[email-masked]>
>>>>>>>>>>>>>> >      >     <mailto:[email-masked]
>>>>>>>>>>>>>> >     <mailto:[email-masked]>>>>>> wrote:
>>>>>>>>>>>>>> >      >      >      >              >
>>>>>>>>>>>>>> >      >      >      >              >     Hi Dinesh,
>>>>>>>>>>>>>> >      >      >      >              >     thank you for your
>>>>>>>>>>>>>> consideration
>>>>>>>>>>>>>> >     of the
>>>>>>>>>>>>>> >      >      >     proposal and
>>>>>>>>>>>>>> >      >      >      >             questions. What
>>>>>>>>>>>>>> >      >      >      >              >     would you see as
>>>>>>>>>>>>>> the scope of
>>>>>>>>>>>>>> >     testing the
>>>>>>>>>>>>>> >      >      >      >             connectivity for the
>>>>>>>>>>>>>> >      >      >      >              >     specific VNI? If it
>>>>>>>>>>>>>> is
>>>>>>>>>>>>>> >      >     tenant-to-tenant, then
>>>>>>>>>>>>>> >      >      >     VTEPs
>>>>>>>>>>>>>> >      >      >      >             will treat these
>>>>>>>>>>>>>> >      >      >      >              >     packets as regular
>>>>>>>>>>>>>> user frames. More
>>>>>>>>>>>>>> >      >     likely, these
>>>>>>>>>>>>>> >      >      >      >             could be Layer 2
>>>>>>>>>>>>>> >      >      >      >              >     OAM, e.g. CCM
>>>>>>>>>>>>>> frames. The reason
>>>>>>>>>>>>>> >     to use
>>>>>>>>>>>>>> >      >     127/8 for
>>>>>>>>>>>>>> >      >      >      >             IPv4, and
>>>>>>>>>>>>>> >      >      >      >              >
>>>>>>>>>>>>>>  0:0:0:0:0:FFFF:7F00:0/104 for
>>>>>>>>>>>>>> >     IPv6 is
>>>>>>>>>>>>>> >      >     to safeguard
>>>>>>>>>>>>>> >      >      >      >             from leaking
>>>>>>>>>>>>>> >      >      >      >              >     Ethernet frames
>>>>>>>>>>>>>> with BFD Control
>>>>>>>>>>>>>> >     packet
>>>>>>>>>>>>>> >      >     to a
>>>>>>>>>>>>>> >      >      >     tenant.
>>>>>>>>>>>>>> >      >      >      >              >     You've suggested
>>>>>>>>>>>>>> using a MAC
>>>>>>>>>>>>>> >     address to
>>>>>>>>>>>>>> >      >     trap the
>>>>>>>>>>>>>> >      >      >      >             control packet at
>>>>>>>>>>>>>> >      >      >      >              >     VTEP. What that
>>>>>>>>>>>>>> address could be? We
>>>>>>>>>>>>>> >      >     had proposed
>>>>>>>>>>>>>> >      >      >      >             using the
>>>>>>>>>>>>>> >      >      >      >              >     dedicated MAC and
>>>>>>>>>>>>>> VTEP's MAC and
>>>>>>>>>>>>>> >     both
>>>>>>>>>>>>>> >      >     raised
>>>>>>>>>>>>>> >      >      >     concerns
>>>>>>>>>>>>>> >      >      >      >             among VXLAN
>>>>>>>>>>>>>> >      >      >      >              >     experts. The idea
>>>>>>>>>>>>>> of using
>>>>>>>>>>>>>> >     Management
>>>>>>>>>>>>>> >      >     VNI may
>>>>>>>>>>>>>> >      >      >     be more
>>>>>>>>>>>>>> >      >      >      >             acceptable
>>>>>>>>>>>>>> >      >      >      >              >     based on its
>>>>>>>>>>>>>> similarity to the
>>>>>>>>>>>>>> >     practice
>>>>>>>>>>>>>> >      >     of using
>>>>>>>>>>>>>> >      >      >      >             Management VLAN.
>>>>>>>>>>>>>> >      >      >      >              >
>>>>>>>>>>>>>> >      >      >      >              >     Regards,
>>>>>>>>>>>>>> >      >      >      >              >     Greg
>>>>>>>>>>>>>> >      >      >      >              >
>>>>>>>>>>>>>> >      >      >      >              >     On Wed, Jul 31,
>>>>>>>>>>>>>> 2019 at 12:03 PM
>>>>>>>>>>>>>> >     Dinesh
>>>>>>>>>>>>>> >      >     Dutt
>>>>>>>>>>>>>> >      >      >      >             <[email-masked]
>>>>>>>>>>>>>> >     <mailto:[email-masked]> <mailto:[email-masked]
>>>>>>>>>>>>>> >     <mailto:[email-masked]>>
>>>>>>>>>>>>>> >      >     <mailto:[email-masked] <mailto:
>>>>>>>>>>>>>> [email-masked]>
>>>>>>>>>>>>>> >     <mailto:[email-masked] <mailto:[email-masked]>>>
>>>>>>>>>>>>>> >      >      >     <mailto:[email-masked] <mailto:
>>>>>>>>>>>>>> [email-masked]>
>>>>>>>>>>>>>> >     <mailto:[email-masked] <mailto:[email-masked]>>
>>>>>>>>>>>>>> >      >     <mailto:[email-masked] <mailto:
>>>>>>>>>>>>>> [email-masked]>
>>>>>>>>>>>>>> >     <mailto:[email-masked] <mailto:[email-masked]>>>>
>>>>>>>>>>>>>> >      >      >      >              >     <mailto:
>>>>>>>>>>>>>> [email-masked]
>>>>>>>>>>>>>> >     <mailto:[email-masked]>
>>>>>>>>>>>>>> >      >     <mailto:[email-masked] <mailto:
>>>>>>>>>>>>>> [email-masked]>>
>>>>>>>>>>>>>> >      >      >     <mailto:[email-masked] <mailto:
>>>>>>>>>>>>>> [email-masked]>
>>>>>>>>>>>>>> >     <mailto:[email-masked] <mailto:[email-masked]>>>
>>>>>>>>>>>>>> >      >     <mailto:[email-masked] <mailto:
>>>>>>>>>>>>>> [email-masked]>
>>>>>>>>>>>>>> >     <mailto:[email-masked] <mailto:[email-masked]>>
>>>>>>>>>>>>>> >      >      >     <mailto:[email-masked] <mailto:
>>>>>>>>>>>>>> [email-masked]>
>>>>>>>>>>>>>> >     <mailto:[email-masked] <mailto:[email-masked]>>>>>>
>>>>>>>>>>>>>> >      >      >      >             wrote:
>>>>>>>>>>>>>> >      >      >      >              >
>>>>>>>>>>>>>> >      >      >      >              >         Hi Greg,
>>>>>>>>>>>>>> >      >      >      >              >
>>>>>>>>>>>>>> >      >      >      >              >         As long as the
>>>>>>>>>>>>>> inner MAC
>>>>>>>>>>>>>> >     address is
>>>>>>>>>>>>>> >      >     such
>>>>>>>>>>>>>> >      >      >     that the
>>>>>>>>>>>>>> >      >      >      >             packet is
>>>>>>>>>>>>>> >      >      >      >              >         trapped to the
>>>>>>>>>>>>>> CPU, it should be
>>>>>>>>>>>>>> >      >     fine for
>>>>>>>>>>>>>> >      >      >     use as
>>>>>>>>>>>>>> >      >      >      >             an inner MAC is
>>>>>>>>>>>>>> >      >      >      >              >         it not? Stating
>>>>>>>>>>>>>> that is
>>>>>>>>>>>>>> >     better than
>>>>>>>>>>>>>> >      >     trying to
>>>>>>>>>>>>>> >      >      >      >             force a management
>>>>>>>>>>>>>> >      >      >      >              >         VNI. What if
>>>>>>>>>>>>>> someone wants
>>>>>>>>>>>>>> >     to test
>>>>>>>>>>>>>> >      >      >     connectivity
>>>>>>>>>>>>>> >      >      >      >             on a specific
>>>>>>>>>>>>>> >      >      >      >              >         VNI? I would
>>>>>>>>>>>>>> not pick a
>>>>>>>>>>>>>> >     loopback IP
>>>>>>>>>>>>>> >      >      >     address for
>>>>>>>>>>>>>> >      >      >      >             this since that
>>>>>>>>>>>>>> >      >      >      >              >         address range
>>>>>>>>>>>>>> is host/node local
>>>>>>>>>>>>>> >      >     only. Is
>>>>>>>>>>>>>> >      >      >     there a
>>>>>>>>>>>>>> >      >      >      >             reason you're
>>>>>>>>>>>>>> >      >      >      >              >         not using the
>>>>>>>>>>>>>> VTEP IP as the
>>>>>>>>>>>>>> >     inner IP
>>>>>>>>>>>>>> >      >      >     address ?
>>>>>>>>>>>>>> >      >      >      >              >
>>>>>>>>>>>>>> >      >      >      >              >         Dinesh
>>>>>>>>>>>>>> >      >      >      >              >
>>>>>>>>>>>>>> >      >      >      >              >         On Wed, Jul 31,
>>>>>>>>>>>>>> 2019 at 5:48 AM
>>>>>>>>>>>>>> >      >     Greg Mirsky
>>>>>>>>>>>>>> >      >      >      >              >         <
>>>>>>>>>>>>>> [email-masked]
>>>>>>>>>>>>>> >     <mailto:[email-masked]>
>>>>>>>>>>>>>> >      >     <mailto:[email-masked] <mailto:
>>>>>>>>>>>>>> [email-masked]>>
>>>>>>>>>>>>>> >      >      >     <mailto:[email-masked]
>>>>>>>>>>>>>> >     <mailto:[email-masked]> <mailto:
>>>>>>>>>>>>>> [email-masked]
>>>>>>>>>>>>>> >     <mailto:[email-masked]>>>
>>>>>>>>>>>>>> >      >      >      >             <mailto:
>>>>>>>>>>>>>> [email-masked]
>>>>>>>>>>>>>> >     <mailto:[email-masked]>
>>>>>>>>>>>>>> >      >     <mailto:[email-masked] <mailto:
>>>>>>>>>>>>>> [email-masked]>>
>>>>>>>>>>>>>> >      >      >     <mailto:[email-masked]
>>>>>>>>>>>>>> >     <mailto:[email-masked]>
>>>>>>>>>>>>>> >      >     <mailto:[email-masked]
>>>>>>>>>>>>>> >     <mailto:[email-masked]>>>> <mailto:
>>>>>>>>>>>>>> [email-masked]
>>>>>>>>>>>>>> >     <mailto:[email-masked]>
>>>>>>>>>>>>>> >      >     <mailto:[email-masked] <mailto:
>>>>>>>>>>>>>> [email-masked]>>
>>>>>>>>>>>>>> >      >      >     <mailto:[email-masked]
>>>>>>>>>>>>>> >     <mailto:[email-masked]> <mailto:
>>>>>>>>>>>>>> [email-masked]
>>>>>>>>>>>>>> >     <mailto:[email-masked]>>>
>>>>>>>>>>>>>> >      >      >      >             <mailto:
>>>>>>>>>>>>>> [email-masked]
>>>>>>>>>>>>>> >     <mailto:[email-masked]>
>>>>>>>>>>>>>> >      >     <mailto:[email-masked] <mailto:
>>>>>>>>>>>>>> [email-masked]>>
>>>>>>>>>>>>>> >      >      >     <mailto:[email-masked]
>>>>>>>>>>>>>> >     <mailto:[email-masked]>
>>>>>>>>>>>>>> >      >     <mailto:[email-masked]
>>>>>>>>>>>>>> >     <mailto:[email-masked]>>>>>> wrote:
>>>>>>>>>>>>>> >      >      >      >              >
>>>>>>>>>>>>>> >      >      >      >              >             Dear All,
>>>>>>>>>>>>>> >      >      >      >              >             thank you
>>>>>>>>>>>>>> for your comments,
>>>>>>>>>>>>>> >      >      >     suggestions on
>>>>>>>>>>>>>> >      >      >      >             this issue,
>>>>>>>>>>>>>> >      >      >      >              >             probably
>>>>>>>>>>>>>> the most
>>>>>>>>>>>>>> >     challenging
>>>>>>>>>>>>>> >      >     for this
>>>>>>>>>>>>>> >      >      >      >             specification. In the
>>>>>>>>>>>>>> >      >      >      >              >             course of
>>>>>>>>>>>>>> our discussions,
>>>>>>>>>>>>>> >      >     we've agreed to
>>>>>>>>>>>>>> >      >      >      >             abandon the
>>>>>>>>>>>>>> >      >      >      >              >             request to
>>>>>>>>>>>>>> allocate the
>>>>>>>>>>>>>> >      >     dedicated MAC
>>>>>>>>>>>>>> >      >      >     address
>>>>>>>>>>>>>> >      >      >      >             to be used as
>>>>>>>>>>>>>> >      >      >      >              >             the
>>>>>>>>>>>>>> destination MAC
>>>>>>>>>>>>>> >     address in
>>>>>>>>>>>>>> >      >     the inner
>>>>>>>>>>>>>> >      >      >      >             Ethernet frame.
>>>>>>>>>>>>>> >      >      >      >              >             Also,
>>>>>>>>>>>>>> earlier using VNI
>>>>>>>>>>>>>> >     0 was
>>>>>>>>>>>>>> >      >     changed from
>>>>>>>>>>>>>> >      >      >      >             mandatory to one
>>>>>>>>>>>>>> >      >      >      >              >             of the
>>>>>>>>>>>>>> options an
>>>>>>>>>>>>>> >      >     implementation may
>>>>>>>>>>>>>> >      >      >     offer to
>>>>>>>>>>>>>> >      >      >      >             an operator.
>>>>>>>>>>>>>> >      >      >      >              >             The most
>>>>>>>>>>>>>> recent
>>>>>>>>>>>>>> >     discussion was
>>>>>>>>>>>>>> >      >     whether
>>>>>>>>>>>>>> >      >      >     VTEP's
>>>>>>>>>>>>>> >      >      >      >             MAC address
>>>>>>>>>>>>>> >      >      >      >              >             might be
>>>>>>>>>>>>>> used as the
>>>>>>>>>>>>>> >      >     destination MAC
>>>>>>>>>>>>>> >      >      >     address
>>>>>>>>>>>>>> >      >      >      >             in the inner
>>>>>>>>>>>>>> >      >      >      >              >             Ethernet
>>>>>>>>>>>>>> frame. As I
>>>>>>>>>>>>>> >     recall it, the
>>>>>>>>>>>>>> >      >      >     comments
>>>>>>>>>>>>>> >      >      >      >             from VXLAN
>>>>>>>>>>>>>> >      >      >      >              >             experts
>>>>>>>>>>>>>> equally split
>>>>>>>>>>>>>> >     with one
>>>>>>>>>>>>>> >      >     for it
>>>>>>>>>>>>>> >      >      >     and one
>>>>>>>>>>>>>> >      >      >      >             against. Hence
>>>>>>>>>>>>>> >      >      >      >              >             I would
>>>>>>>>>>>>>> like to propose
>>>>>>>>>>>>>> >     a new
>>>>>>>>>>>>>> >      >     text to
>>>>>>>>>>>>>> >      >      >     resolve
>>>>>>>>>>>>>> >      >      >      >             the issue. The
>>>>>>>>>>>>>> >      >      >      >              >             idea is to
>>>>>>>>>>>>>> let an
>>>>>>>>>>>>>> >     operator select
>>>>>>>>>>>>>> >      >      >     Management
>>>>>>>>>>>>>> >      >      >      >             VNI and use
>>>>>>>>>>>>>> >      >      >      >              >             that VNI in
>>>>>>>>>>>>>> VXLAN
>>>>>>>>>>>>>> >     encapsulation
>>>>>>>>>>>>>> >      >     of BFD
>>>>>>>>>>>>>> >      >      >      >             Control packets:
>>>>>>>>>>>>>> >      >      >      >              >             NEW TEXT:
>>>>>>>>>>>>>> >      >      >      >              >
>>>>>>>>>>>>>> >      >      >      >              >                 An
>>>>>>>>>>>>>> operator MUST
>>>>>>>>>>>>>> >     select a VNI
>>>>>>>>>>>>>> >      >      >     number to
>>>>>>>>>>>>>> >      >      >      >             be used as
>>>>>>>>>>>>>> >      >      >      >              >
>>>>>>>>>>>>>>  Management VNI. VXLAN
>>>>>>>>>>>>>> >      >     packet for
>>>>>>>>>>>>>> >      >      >      >             Management VNI MUST NOT
>>>>>>>>>>>>>> >      >      >      >              >                 be sent
>>>>>>>>>>>>>> to a tenant. VNI
>>>>>>>>>>>>>> >      >     number 1 is
>>>>>>>>>>>>>> >      >      >      >             RECOMMENDED as the
>>>>>>>>>>>>>> >      >      >      >              >                 default
>>>>>>>>>>>>>> for
>>>>>>>>>>>>>> >     Management VNI.
>>>>>>>>>>>>>> >      >      >      >              >
>>>>>>>>>>>>>> >      >      >      >              >             With that
>>>>>>>>>>>>>> new text, what
>>>>>>>>>>>>>> >     can be the
>>>>>>>>>>>>>> >      >      >     value of
>>>>>>>>>>>>>> >      >      >      >             the destination
>>>>>>>>>>>>>> >      >      >      >              >             MAC in the
>>>>>>>>>>>>>> inner Ethernet? I
>>>>>>>>>>>>>> >      >     tend to
>>>>>>>>>>>>>> >      >      >     believe
>>>>>>>>>>>>>> >      >      >      >             that it can be
>>>>>>>>>>>>>> >      >      >      >              >             anything
>>>>>>>>>>>>>> and ignored by the
>>>>>>>>>>>>>> >      >     reciever VTEP.
>>>>>>>>>>>>>> >      >      >      >             Also, if the
>>>>>>>>>>>>>> >      >      >      >              >             trapping is
>>>>>>>>>>>>>> based on VNI
>>>>>>>>>>>>>> >      >     number, the
>>>>>>>>>>>>>> >      >      >      >             destination IP address
>>>>>>>>>>>>>> >      >      >      >              >             of the
>>>>>>>>>>>>>> inner IP packet
>>>>>>>>>>>>>> >     can from
>>>>>>>>>>>>>> >      >     the range
>>>>>>>>>>>>>> >      >      >      >             127/8 for IPv4,
>>>>>>>>>>>>>> >      >      >      >              >             and for
>>>>>>>>>>>>>> IPv6 from the range
>>>>>>>>>>>>>> >      >      >      >             0:0:0:0:0:FFFF:7F00:0/104.
>>>>>>>>>>>>>> And
>>>>>>>>>>>>>> >      >      >      >              >             lastly, the
>>>>>>>>>>>>>> TTL to be
>>>>>>>>>>>>>> >     set to 1 (no
>>>>>>>>>>>>>> >      >      >     change here).
>>>>>>>>>>>>>> >      >      >      >              >
>>>>>>>>>>>>>> >      >      >      >              >             Much
>>>>>>>>>>>>>> appreciate your
>>>>>>>>>>>>>> >     comments,
>>>>>>>>>>>>>> >      >      >     questions, and
>>>>>>>>>>>>>> >      >      >      >             suggestions.
>>>>>>>>>>>>>> >      >      >      >              >
>>>>>>>>>>>>>> >      >      >      >              >             Best
>>>>>>>>>>>>>> regards,
>>>>>>>>>>>>>> >      >      >      >              >             Greg
>>>>>>>>>>>>>> >      >      >      >              >
>>>>>>>>>>>>>> >      >      >      >
>>>>>>>>>>>>>> >      >      >
>>>>>>>>>>>>>> >      >
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>
>>>>>>>>>>>>>

---

From: Santosh P K <[email-masked]>
Date: Sun, 22 Sep 2019 13:11:20 +0530

Greg,
    Please see inline reply tagged [SPK]. I have added text requested.

Thanks
Santosh P K

> GIM>> Certainly. Please share the text you'd like to add.

[SPK]  Proposed text for why we should not use VTEP MAX address as tenant
MAC address.

"In some scenarios tenant MAC is borrowed from VTEP MAC address. VXLAN BFD
MUST terminate BFD session at VTEP and MUST not forward BFD packets to
tenants. To terminate VXLAN BFD packets at VTEP, deployment MUST ensure
that there are no tenant VM which barrows VTEP MAC address."

>          packet as VXLAN  gateway address is routable address.

[SPK] This text looks good.

> What is the goal running multiple BFD sessions between the pair of VTEPs?
[SPK] The goal of the multiple BFD session is to ensure check liveliness of
VXLAN tunnel. There is already a good amount of debate on this topic that
do we really need that? As per RFC 5884 we are running BFD per LSP and we
might hit scale issues there too. I think it is up to operator to decide
how they want to use multiple BFD session per VXLAN tunnel. It could be
possible that BFD session with special VNI is run at aggressive interval
where as MAY have multiple BFD sessions for different VNI at a sedate
interval, for that matter they could be running in demand mode as well (run
P/F sequence only when there is no data following for that VNI). As WG if
we think running multiple BFD session make sense then we might need to add
appropriate text.

>    MUST be demultiplexed only with Your Discriminator as the key.

[SPK]  I think this text for multiple BFD session between same pair of
VTEPs for multiple VNI makes sense only if as WG we think that could be use
case.

[SPK] IMHO we should not allow multiple BFD session for the same VNI.

---

From: <[email-masked]>
Date: Wed, 25 Sep 2019 10:39:41 +0800 (CST)

Hi Santosh,

With regard to the question whether we should allow multiple BFD sessions for the same VNI or not, IMHO we should allow it, more explanation as follows.

Below is a figure derived from figure 2 of RFC8014 (An Architecture for Data-Center Network Virtualization over Layer 3 (NVO3)).

 | Data Center Network (IP) |
 | |
 +-----------------------------------------+
 | |
 | Tunnel Overlay |
 +------------+---------+ +---------+------------+
 | +----------+-------+ | | +-------+----------+ |
 | | Overlay Module | | | | Overlay Module | |
 | +---------+--------+ | | +---------+--------+ |
 | | | | | |
 NVE1 | | | | | | NVE2
 | +--------+-------+ | | +--------+-------+ |
 | |VNI1 VNI2 VNI1 | | | | VNI1 VNI2 VNI1 | |
 | +-+-----+----+---+ | | +-+-----+-----+--+ |
 |VAP1| VAP2| | VAP3 | |VAP1| VAP2| | VAP3|
 +----+-----+----+------+ +----+-----+-----+-----+
 | | | | | |
 | | | | | |
 | | | | | |
 -------+-----+----+-------------------+-----+-----+-------
 | | | Tenant | | |
 TSI1 | TSI2| | TSI3 TSI1| TSI2| |TSI3
 +---+ +---+ +---+ +---+ +---+ +---+
 |TS1| |TS2| |TS3| |TS4| |TS5| |TS6|
 +---+ +---+ +---+ +---+ +---+ +---+
To my understanding, the BFD sessions between NVE1 and NVE2 are actually initiated and terminated at VAP of NVE.

If the network operator want to set up one BFD session between VAP1 of NVE1 and VAP1of NVE2, at the same time another BFD session between VAP3 of NVE1 and VAP3 of NVE2, although the two BFD sessions are for the same VNI1, I believe it's reasonable, so that's why I think we should allow it.

Of course, in RFC8014 it also says:

"Note that two different Tenant Systems (and TSIs) attached to a common NVE can share a VAP (e.g., TS1 and TS2 in Figure 2) so long as they connect to the same Virtual Network."
Some people may argue that all Tenant Systems connecting to the same Virtual Network MUST share one VAP, if that's true, then VAP1 and VAP3 should merge into one VAP and my explanation doesn't work. Copying to NVO3 WG to involve more experts, hope for your clarifications and comments.

Best Regards,

Xiao Min

原始邮件

发件人：SantoshPK <[email-masked]>
收件人：Greg Mirsky <[email-masked]>;
抄送人：[email-masked] <[email-masked]>;Dinesh Dutt <[email-masked]>;rtg-bfd WG <[email-masked]>;Joel M. Halpern <[email-masked]>;T. Sridhar <[email-masked]>;[email-masked] <[email-masked]>;
日 期 ：2019年09月23日 05:39
主 题 ：Re: BFD over VXLAN: Trapping BFD Control packet at VTEP

Greg,

Thanks
Santosh P K 

On Fri, Aug 16, 2019 at 4:59 AM Greg Mirsky <gregimirsky@gmail...com> wrote:

Hi Santosh,thank you for your comments. Please find my notes in-lined and tagged GIM>>.

Regards,
Greg

On Tue, Aug 13, 2019 at 10:24 PM Santosh P K <[email-masked]> wrote:

Greg,   Thanks for updated version of document. Here are few comments on new draft.

Section 4:

I think we may need to give background on why we are saying MAC address MUST not be one of tenant's MAC address. Like in this thread we have discussed one of the tenant could have borrowed the same VTEP mac address and we if we have to use BFD then we need to avoid that conflict to ensure BFD packets get observed in the VTEP itself. Should we add a section before 4 to set that context so that above text makes more sense in that context?

 [SPK]  Proposed text for why we should not use VTEP MAX address as tenant MAC address.

"In some scenarios tenant MAC is borrowed from VTEP MAC address. VXLAN BFD MUST terminate BFD session at VTEP and MUST not forward BFD packets to tenants. To terminate VXLAN BFD packets at VTEP, deployment MUST ensure that there are no tenant VM which barrows VTEP MAC address."

   IP header:

I think we have added some text to address Sridhar comments on why TTL MUST be 1 and dest IP address MUST be 127/8 range. I see that text is missing now.

GIM>> My apologies that I've missed to include the text from another discussion thread. I believe the following would be complete:

Section 5.1:

Just with 3 tuple we will not be able to demux packet. We need to consider VNI as well if we have multiple BFD session between same pair of VTEP.

GIM>> This is one of comments from Carlos we need to address. Your comment have helped me to form the question:

 [SPK] The goal of the multiple BFD session is to ensure check liveliness of VXLAN tunnel. There is already a good amount of debate on this topic that do we really need that? As per RFC 5884 we are running BFD per LSP and we might hit scale issues there too. I think it is up to operator to decide how they want to use multiple BFD session per VXLAN tunnel. It could be possible that BFD session with special VNI is run at aggressive interval where as MAY have multiple BFD sessions for different VNI at a sedate interval, for that matter they could be running in demand mode as well (run P/F sequence only when there is no data following for that VNI). As WG if we think running multiple BFD session make sense then we might need to add appropriate text.

If the goal is to monitor per VNI, then the following text should describe the demultiplexing of the initial BFD Control packet:   Demultiplexing of IP BFD packet has been defined in Section 3 of

 [SPK]  I think this text for multiple BFD session between same pair of VTEPs for multiple VNI makes sense only if as WG we think that could be use case. 

Thanks

Santosh P K

Dinesh, thank you for your help, much appreciated.
Hi Joel and Sridhar,
could you please check if the updated text on the inner Ethernet frame addressed your concern.

Dinesh

On Wed, Aug 7, 2019 at 9:25 AM Greg Mirsky <[email-masked]> wrote:

Hi Dinesh, Joel, Sridhar, et al.,much appreciate the help you've given me sharing your expertise. I hope that the updates you will find in the attached diff and the working copy of the draft be closer to the acceptable solution for VTEP-VTEP BFD. Please note, that I'll shortly start a new discussion thread to address one of Carlos's questions on the ambiguity of the text on multiple concurrent sessions between the same pair of VTEPs.
Please review the changes to Sections 4 and 6 and share your feedback, suggestions, and questions.

Regards,
Greg

On Mon, Aug 5, 2019 at 5:56 PM Greg Mirsky <[email-masked]> wrote:

Hi Dinesh,thank you for your expedient detailed response.
I believe that the ability to run BFD session up to a tenant (VTEP-VTEP-tenant or tenant-tenant) was never in jeopardy from this specification.
I'm trying to provide precise specification on what can be used ad the destination MAC and IP addresses in the inner frame/packet as I believe that likely will help to avoid interoperability issues.
I'm interested to learn some more about the "martian checking" function. As you know, martian addresses have been used as destination IP address in LSP Ping and BFD over MPLS LSP and PW. I haven't heard that any silicon feature caused problems for operators using these tools.

Interesting. I didn't know this aspect of use with MPLS ping. Did those packets ever go through a firewall though? In any case, maybe suggest the use of those addresses with a statement that this is how LSP does it, but that other MAC/IP pairs are possible as long as the conditions of the endpoint owning the MAC/IP was honored.

Dinesh

Regards,
Greg

Hi Greg,

That we agree on the problem definition is the first step forward. Your original document had my cases covered and so I was surprised by the track this thread took. It doesn't matter, we're back on track.

My recommendation is to not worry about specifying the precise MAC/IP address used in the inner header. The addresses chosen MUST ensure that the packet is trapped to the control plane of the VTEP and not escape to the tenant if the BFD is to the VTEP. Any solution MUST also not preclude the use of the BFD by tenant systems for that VNI. There are many ways an implementer can choose to implement this. For example, the inner MAC address is whatever the VTEP implementer would return if ARP'd for the IP address used in the inner header in the given VNI. The implementer can pick a fixed MAC address, one that they own etc. Multiple BFD sessions can be run for testing path connectivity on more than one VNIs. Limits should be in place to avoid overwhelming the receiver with BFD messages (you had words about this in your currently published draft).  If the VNI is irrelevant in the test i.e. only the VXLAN pipe at the VTEP is being tested. the user can use any VNI active on the VTEP on which the VTEP owns an IP address.

I'm concerned about the use of 127/8 address only because of firewalls or implementations that drop packets with these addresses as either the source or destination. For example, on many merchant silicon, I don't believe you can turn off martian checking and drops *only* for VXLAN-encapsulated BFD packets. I don't know what the Linux kernel does today on such packets, for example (or Hyper-V). I'd like a solution that doesn't demand additional or new chip functionality or require additional middle-box hole punch.

Why do you feel you MUST to specify the MAC/IP address on the inner packet? What am I missing here?

Dinesh

On Mon, Aug 5, 2019 at 3:04 PM Greg Mirsky <[email-masked]> wrote:

Hi Dinesh,what do you see as the way forward? I agree, that the proposed text doesn't work for multi-VNI concurrent monitoring because these VNIs are tenant's VNIs. And in that case, we need to specify another mechanism to trap the BFD Control packet at VTEP. It seems that VTEP's Ethernet address must be used as the destination MAC address in the inner Ethernet frame. The destination IP address may be either VTEP's address of martian (I do prefer martian). Let me give it  try:
NEW TEXT:

To monitor continuity of the path between two VTEPs, an operator MUST select a VNI number to be used as Management VNI. Management VNI number MUST NOT be one of the tenant's VNIs to prevent sending VXLAN packets received on Management VNI to a tenant. VNI number 1 is RECOMMENDED as the default for Management VNI. [Ed.note: What we set the Destination MAC to? Can it be invalid MAC that MUST be ignored on receipt?]

If an implementation supports concurrent monitoring of multiple VNIs, then the value of VNI number MAY be one of tenant's VNIs. The destination MAC address in the inner Ethernet frame encapsulating BFD Control packet MUST be MAC associated with the remote VTEP. 
The destination IP address of the inner IP packet MUST be selected from the range 127/8 for IPv4, and for IPv6 from the range 0:0:0:0:0:FFFF:7F00:0/104. The TTL value in the inner IP header MUST be set to 1.

Regards,Greg 

On Sun, Aug 4, 2019 at 9:07 AM Dinesh Dutt <[email-masked]> wrote:

Hi Greg,

Thanks for your clarifications. I agree with your sentiment on why you're running BFD over VXLAN between VTEPs. I wasn't arguing against it at all. All I was saying was pointing to the limitations of the use of management VNI. I spoke to some operators who're running EVPN and mentioned the discussion on this thread. They concur that they're using specific VNIs to test connectivity over that VNI between VTEPs to ensure misconfiguration doesn't lead to blackholes. My statements are based in real world operator experience. And I was providing language that ensured packets didn't leak across to tenants when they were destined to VTEPs.

Dinesh

On Sat, Aug 3, 2019 at 10:34 AM Greg Mirsky <[email-masked]> wrote:

Hi Dinesh,many thanks for your detailed updates on how some implementations process VXLAN header and the inner Ethernet frame. These are very helpful in achieving the workable solution for the problem at hand.
You've noted that a path between VTEPs may be monitored in the underlay network by merely establishing a BFD session. That is true, but by using BFD with VXLAN encapsulation between the pair of VTEPs we are extending the OAM domain by including, to some extent, VXLAN forwarding engine. Abstract in RFC 5880 defines the goal and the domain in which BFD protocol can detect a fault as:
   This document describes a protocol intended to detect faults in the
   bidirectional path between two forwarding engines, including
   interfaces, data link(s), and to the extent possible the forwarding
   engines themselves, with potentially very low latency. 

Thus, BFD in the underlay will exercise a part of IP forwarding engine while BFD with VXLAN encapsulation, ran between the same pair of VTEPs, extends the OAM domain. At the same time, defining BFD between tenant systems in outside the goal of this specification. But VXLAN BFD session between VTEPs may be useful in monitoring e2e path between tenants, as described in the update to -07:
   At the same time, a service layer BFD session may be used between the
   tenants of VTEPs IP1 and IP2 to provide end-to-end fault management.
   In such case, for VTEPs BFD control packets of that session are
   indistinguishable from data packets.  If end-to-end defect detection
   is realized as the set of concatenated OAM domains, e.g., VM1-1 - IP1
   -- IP2 - VM2-1, then the BFD session over VXLAN between VTEPs SHOULD
   follow the procedures described in Section 6.8.17 [RFC5880].

Regards,
Greg

On Fri, Aug 2, 2019 at 5:43 PM Dinesh Dutt <[email-masked]> wrote:

What I mean is "How do you infer that it excludes the case I'm talking about?". 

Dinesh

On Fri, Aug 2, 2019 at 5:41 PM Dinesh Dutt <[email-masked]> wrote:

Dinesh

On Fri, Aug 2, 2019 at 5:38 PM Joel M. Halpern <[email-masked]> wrote:

I am going by what the draft says its purpose is.  If you (Dinesh) want 
the draft to fulfill a different purpose, then either ask the chairs to 
take this draft back to the WG, or write a separate draft.
As currently written, the behavior Greg proposed meets the needs, and 
does so in a way that is consistent with VxLAN.

Yours,
Joel

On 8/2/2019 8:30 PM, Dinesh Dutt wrote:
> What is the stated purpose of this BFD session? The VTEP reachability is 
> determined by the underlay, I don't need VXLAN-encaped packet for that. 
> Do we agree?
> 
> If I want to test the VXLAN encap/decap functionality alone, picking any 
> single VNI maybe fine. But is this all any network operator wants? Why? 
> In what situations has this been a problem? I suspect operators also 
> want to verify path continuity over a specific VNI. If you say this is 
> not defined by the document, I disagree because the current version 
> talks about controlling the number of BFD sessions between the VTEPs 
> (see section 3). More importantly, this is a real problem that operators 
> like to verify.
> 
> Dinesh
> 
> On Fri, Aug 2, 2019 at 5:08 PM Joel M. Halpern <[email-masked] 
>     What is special about the management VNI is precisely that it is NOT a
>     tenant VNI.  The VxLAN administration does know how it allocates VNI to
>     tenants, and which VNI it has allocated.  In contrast, it does not know
>     which IP addresses or MAC adddresses teh tenant is using or may plan
>     to use.
> 
>     Yours,
>     Joel
> 
>      > The assumption of an IP address within any VNI is suspect that way.
>      > What's special about a single VNI, the management VNI? The VTEP IP
>      > address does not belong in reality in any VNI...
>      >
>      > Dinesh
>      >
>      > <mailto:[email-masked] <mailto:[email-masked]>>> wrote:
>      >
>      >     First, the problem you describe is not what the document says
>     it is
>      >     solving.  To the degree it discusses it at all, the document
>     says "
>      >       In
>      >     most cases, a single BFD session is sufficient for the given
>     VTEP to
>      >     monitor the reachability of a remote VTEP, regardless of the
>     number of
>      >     VNIs in common. "
>      >
>      >     Second, you assume the existence of an IP address for a VTEP
>     within a
>      >     VNI.  As with the MAC address, the VTEP does not have an IP
>     address
>      >     within the VNI.  Some implementations may have created such a
>     thing,
>      >     but
>      >     the general construct, as defined to date, does not support such.
>      >
>      >     In short, you are requiring a behavior that violates the
>     architectural
>      >     structure of overlay / underlay separation, and common
>     usage.  And you
>      >     are doing so to support a use case that the working group has not
>      >      > The VNIs may not share fate due to misconfiguration. And I
>     strongly
>      >      > suspect someone will want to use BFD for that because its
>     about
>      >     checking
>      >      > path continuity as stated by the draft. As long as there's a
>      >     valid IP
>      >      > (because it's BFD) owned by the VTEP in that VNI, you can
>     use BFD in
>      >      > that VNI. Thats all that you need to dictate.  That IP address
>      >     has a MAC
>      >      > address and you can use that on the inner frame. That is
>     all normal
>      >      > VXLAN processing. The outer IP is always that of the VTEP.
>      >      >
>      >      > Dinesh
>      >      >
>      >      > On Fri, Aug 2, 2019 at 11:03 AM Joel M. Halpern
>     <mailto:[email-masked] <mailto:[email-masked]>>
>      >      > <mailto:[email-masked] <mailto:[email-masked]>
>     <mailto:[email-masked] <mailto:[email-masked]>>>> wrote:
>      >      >
>      >      >     If I am reading your various emails correctly Dinesh
>     (and I
>      >     may have
>      >      >     missed something) you are trying to use the MAC address
>      >     because you
>      >      >     want
>      >      >     to be able to send these BFD packets over arbitrary VNI to
>      >     monitor the
>      >      >     VNI.  That is not a requirement identified in the
>     document.
>      >     It is not
>      >      >     even a problem I understand, since all the VNI between an
>      >     ingress and
>      >      >     egress VTEP share fate.
>      >      >
>      >      >     Yours,
>      >      >     Joel
>      >      >
>      >      >      > Thanks for verifying this. On Linux and hardware
>     routers
>      >     that I'm
>      >      >     aware
>      >      >      > of (Cisco circa 2012 and Cumulus), the physical MAC
>     address is
>      >      >     reused
>      >      >      > across the VNIs on the VTEP. Did you check on a non-VMW
>      >     device?
>      >      >     This is
>      >      >      > To address the general case, can we not define a
>      >     well-known (or
>      >      >     reserve
>      >      >      > one) unicast MAC address for use with VTEP? If the MAC
>      >     address is
>      >      >      > configurable in BFD command, this can be moot.
>      >      >      >
>      >      >      > Dinesh
>      >      >      >
>      >      >      > On Fri, Aug 2, 2019 at 10:27 AM Santosh P K
>      >     <mailto:santosh.pallagatti@gmail...com
>      >     <mailto:santosh.pallagatti@gmail...com
>      >     <mailto:santosh.pallagatti@gmail...com
>      >     <mailto:santosh.pallagatti@gmail...com
>      >      >      >     I have cross checked point raised about MAC address
>      >     usage. It is
>      >      >      >     possible that tenant could be using physical MAC
>      >     address and
>      >      >     when a
>      >      >      >     packet comes with valid VNI with a MAC address
>     that is
>      >     being
>      >      >     used by
>      >      >      >     tenant then packet will be sent to that tenant.
>     This rules
>      >      >     out the
>      >      >      >     fact that we could use physical MAC address as
>     inner
>      >     MAC to
>      >      >     ensure
>      >      >      >     packets get terminated at VTEP itself.
>      >      >      >
>      >      >      >     Thanks
>      >      >      >     Santosh P K
>      >      >      >
>      >      >      >     On Wed, Jul 31, 2019 at 11:00 AM Santosh P K
>      >     <mailto:santosh.pallagatti@gmail...com
>      >     <mailto:santosh.pallagatti@gmail...com
>      >     <mailto:santosh.pallagatti@gmail...com
>      >     <mailto:santosh.pallagatti@gmail...com
>      >      >      >             Thanks for your inputs. I checked
>      >     implementation within
>      >      >      >         Vmware. Perhaps I should have been more clear
>      >     about MAC
>      >      >     address
>      >      >      >         space while checking internally. I will cross
>      >     check again for
>      >      >      >         the same and get back on this list.
>      >      >      >
>      >      >      >         Thanks
>      >      >      >         Santosh P K
>      >      >      >
>      >      >      >         On Wed, Jul 31, 2019 at 10:54 AM Joel M.
>     Halpern
>      >      >      >         <[email-masked]
>     <mailto:[email-masked]> <mailto:[email-masked]
>      >     <mailto:[email-masked] <mailto:[email-masked]>
>     <mailto:[email-masked] <mailto:[email-masked]>>>
>     <mailto:[email-masked]> <mailto:[email-masked]
>      >     <mailto:[email-masked] <mailto:[email-masked]>
>     <mailto:[email-masked] <mailto:[email-masked]>>>>> wrote:
>      >      >      >
>      >      >      >             Sorry to ask a stupid question.  Whose
>      >     implementation?
>      >      >      >
>      >      >      >             The reason I ask is that as far as I
>     can tell,
>      >     since the
>      >      >      >             tenant does not
>      >      >      >             have any control access to the VTEP,
>     there is no
>      >      >     reason for
>      >      >      >             the VTEP to
>      >      >      >             have a MAC address in the tenant
>     space.  Yes, the
>      >      >     device has
>      >      >      >             a physical
>      >      >      >             MAC address.  But the tenant could well be
>      >     using that MAC
>      >      >      >             address.  Yes,
>      >      >      >             they would be violating the Ethernet spec.
>      >     But the whole
>      >      >      >             point of
>      >      >      >             segregation is not to care about such
>     issues.
>      >      >      >
>      >      >      >             On the other hand, if you tell me that
>     the VMWare
>      >      >      >             implementation has an
>      >      >      >             Ethernet address that is part of the tenant
>      >     space, well,
>      >      >      >             they made up
>      >      >      >             this particular game.
>      >      >      >
>      >      >      >             Yours,
>      >      >      >             Joel
>      >      >      >
>      >      >      >             On 7/31/2019 1:44 PM, Santosh P K wrote:
>      >      >      >              > I have checked with implementation
>     in data
>      >     path.
>      >      >     When we
>      >      >      >             receive a
>      >      >      >              > packet with valid VNI then lookup
>     for MAC will
>      >      >     happen and
>      >      >      >             it is VTEP own
>      >      >      >              > MAC then it will be trapped to control
>      >     plane for
>      >      >      >             processing. I think we
>      >      >      >              > can have following options
>      >      >      >              > 1. Optional managment VNI
>      >      >      >              > 2. Mandatory inner MAC set to VTEP mac
>      >      >      >              > 3. Inner IP TTL set to 1 to avoid
>      >     forwarding of packet
>      >      >      >             via inner IP
>      >      >      >              > address.
>      >      >      >              >
>      >      >      >              >
>      >      >      >              > Thoughts?
>      >      >      >              >
>      >      >      >              > Thansk
>      >      >      >              > Santosh P K
>      >      >      >              >
>      >      >      >              > On Wed, Jul 31, 2019 at 9:20 AM Greg
>     Mirsky
>      >      >      >             <[email-masked]
>      >     <mailto:[email-masked] <mailto:[email-masked]>>
>     <mailto:[email-masked] <mailto:[email-masked]>
>      >     <mailto:[email-masked] <mailto:[email-masked]>>>
>     <mailto:[email-masked]> <mailto:[email-masked]
>      >     <mailto:[email-masked] <mailto:[email-masked]>
>     <mailto:[email-masked] <mailto:[email-masked]>>>>
>      >     <mailto:[email-masked] <mailto:[email-masked]>>
>     <mailto:[email-masked]> <mailto:[email-masked]
>      >     <mailto:[email-masked] <mailto:[email-masked]>>
>      >      >      >              >     thank you for your consideration
>     of the
>      >      >     proposal and
>      >      >      >             questions. What
>      >      >      >              >     would you see as the scope of
>     testing the
>      >      >      >             connectivity for the
>      >      >      >              >     specific VNI? If it is
>      >     tenant-to-tenant, then
>      >      >     VTEPs
>      >      >      >             will treat these
>      >      >      >              >     packets as regular user frames. More
>      >     likely, these
>      >      >      >             could be Layer 2
>      >      >      >              >     OAM, e.g. CCM frames. The reason
>     to use
>      >     127/8 for
>      >      >      >             IPv4, and
>      >      >      >              >     Ethernet frames with BFD Control
>     packet
>      >     to a
>      >      >     tenant.
>      >      >      >              >     You've suggested using a MAC
>     address to
>      >     trap the
>      >      >      >             control packet at
>      >      >      >              >     VTEP. What that address could be? We
>      >     had proposed
>      >      >      >             using the
>      >      >      >              >     dedicated MAC and VTEP's MAC and
>     both
>      >     raised
>      >      >     concerns
>      >      >      >             among VXLAN
>      >      >      >              >     experts. The idea of using
>     Management
>      >     VNI may
>      >      >     be more
>      >      >      >             acceptable
>      >      >      >              >     based on its similarity to the
>     practice
>      >     of using
>      >      >      >             Management VLAN.
>      >      >      >              >
>      >      >      >              >     Regards,
>      >      >      >              >     Greg
>      >      >      >              >
>      >      >      >              >     On Wed, Jul 31, 2019 at 12:03 PM
>     Dinesh
>      >     Dutt
>      >      >      >             <[email-masked]
>      >     <mailto:[email-masked] <mailto:[email-masked]>
>      >      >     <mailto:[email-masked] <mailto:[email-masked]>
>      >     <mailto:[email-masked] <mailto:[email-masked]>
>      >      >      >              >     <mailto:[email-masked]
>      >      >     <mailto:[email-masked] <mailto:[email-masked]>
>      >     <mailto:[email-masked] <mailto:[email-masked]>
>      >      >     <mailto:[email-masked] <mailto:[email-masked]>
>      >      >      >              >         As long as the inner MAC
>      >      >      >              >         trapped to the CPU, it should be
>      >     fine for
>      >      >     use as
>      >      >      >             an inner MAC is
>      >      >      >              >         it not? Stating that is
>     better than
>      >     trying to
>      >      >      >             force a management
>      >      >      >              >         VNI. What if someone wants
>     to test
>      >      >     connectivity
>      >      >      >             on a specific
>      >      >      >              >         VNI? I would not pick a
>     loopback IP
>      >      >     address for
>      >      >      >             this since that
>      >      >      >              >         address range is host/node local
>      >     only. Is
>      >      >     there a
>      >      >      >             reason you're
>      >      >      >              >         not using the VTEP IP as the
>     inner IP
>      >      >     address ?
>      >      >      >              >
>      >      >      >              >         Dinesh
>      >      >      >              >
>      >      >      >              >         On Wed, Jul 31, 2019 at 5:48 AM
>      >     Greg Mirsky
>      >      >      >              >         <[email-masked]
>      >     <mailto:[email-masked] <mailto:[email-masked]>>
>     <mailto:[email-masked]> <mailto:[email-masked]
>      >     <mailto:[email-masked] <mailto:[email-masked]>>
>     <mailto:[email-masked]>>>> <mailto:[email-masked]
>      >     <mailto:[email-masked] <mailto:[email-masked]>>
>     <mailto:[email-masked]> <mailto:[email-masked]
>      >     <mailto:[email-masked] <mailto:[email-masked]>>
>      >      >      >              >             thank you for your comments,
>      >      >     suggestions on
>      >      >      >             this issue,
>      >      >      >              >             probably the most
>     challenging
>      >     for this
>      >      >      >             specification. In the
>      >      >      >              >             course of our discussions,
>      >     we've agreed to
>      >      >      >             abandon the
>      >      >      >              >             request to allocate the
>      >     dedicated MAC
>      >      >     address
>      >      >      >             to be used as
>      >      >      >              >             the destination MAC
>     address in
>      >     the inner
>      >      >      >             Ethernet frame.
>      >      >      >              >             Also, earlier using VNI
>     0 was
>      >     changed from
>      >      >      >             mandatory to one
>      >      >      >              >             of the options an
>      >     implementation may
>      >      >     offer to
>      >      >      >             an operator.
>      >      >      >              >             The most recent
>     discussion was
>      >     whether
>      >      >     VTEP's
>      >      >      >             MAC address
>      >      >      >              >             might be used as the
>      >     destination MAC
>      >      >     address
>      >      >      >             in the inner
>      >      >      >              >             Ethernet frame. As I
>     recall it, the
>      >      >     comments
>      >      >      >             from VXLAN
>      >      >      >              >             experts equally split
>     with one
>      >     for it
>      >      >     and one
>      >      >      >             against. Hence
>      >      >      >              >             I would like to propose
>     a new
>      >     text to
>      >      >     resolve
>      >      >      >             the issue. The
>      >      >      >              >             idea is to let an
>     operator select
>      >      >     Management
>      >      >      >             VNI and use
>      >      >      >              >             that VNI in VXLAN
>     encapsulation
>      >     of BFD
>      >      >      >             Control packets:
>      >      >      >              >             NEW TEXT:
>      >      >      >              >
>      >      >      >              >                 An operator MUST
>     select a VNI
>      >      >     number to
>      >      >      >             be used as
>      >      >      >              >                 Management VNI. VXLAN
>      >     packet for
>      >      >      >             Management VNI MUST NOT
>      >      >      >              >                 be sent to a tenant. VNI
>      >     number 1 is
>      >      >      >             RECOMMENDED as the
>      >      >      >              >                 default for
>     Management VNI.
>      >      >      >              >
>      >      >      >              >             With that new text, what
>     can be the
>      >      >     value of
>      >      >      >             the destination
>      >      >      >              >             MAC in the inner Ethernet? I
>      >     tend to
>      >      >     believe
>      >      >      >             that it can be
>      >      >      >              >             anything and ignored by the
>      >     reciever VTEP.
>      >      >      >             Also, if the
>      >      >      >              >             trapping is based on VNI
>      >     number, the
>      >      >      >             destination IP address
>      >      >      >              >             of the inner IP packet
>     can from
>      >     the range
>      >      >      >             127/8 for IPv4,
>      >      >      >              >             and for IPv6 from the range
>      >      >      >             0:0:0:0:0:FFFF:7F00:0/104. And
>      >      >      >              >             lastly, the TTL to be
>     set to 1 (no
>      >      >     change here).
>      >      >      >              >
>      >      >      >              >             Much appreciate your
>     comments,
>      >      >     questions, and
>      >      >      >             suggestions.
>      >      >      >              >
>      >      >      >              >             Best regards,
>      >      >      >              >             Greg
>      >      >      >              >
>      >      >      >
>      >      >
>      >
>

---

From: "Joel M. Halpern" <[email-masked]>
Date: Tue, 24 Sep 2019 22:59:13 -0400

As far as I can tell, the current document we have in front of us is 
explicit that the messages are originated and terminated at the VNI.  If 
you want some other behavior, then we need a document that describes 
that behaviors.

Yours,
Joel

On 9/24/2019 10:39 PM, [email-masked] wrote:
> Hi Santosh,
> 
> 
> With regard to the question whether we should allow multiple BFD 
> sessions for the same VNI or not, IMHO we should allow it, more 
> explanation as follows.
> 
> Below is a figure derived from figure 2 of RFC8014 (An Architecture for 
> Data-Center Network Virtualization over Layer 3 (NVO3)).
> 
> If the network operator want to set up one BFD session between VAP1 of 
> NVE1 and VAP1of NVE2, at the same time another BFD session between VAP3 
> of NVE1 and VAP3 of NVE2, although the two BFD sessions are for the same 
> VNI1, I believe it's reasonable, so that's why I think we should allow it.
> 
> 
> Some people may argue that all Tenant Systems connecting to the same 
> Virtual Network MUST share one VAP, if that's true, then VAP1 and VAP3 
> should merge into one VAP and my explanation doesn't work. Copying to 
> NVO3 WG to involve more experts, hope for your clarifications and comments.
> 
> 
> Best Regards,
> 
> Xiao Min
> 
> 原始邮件
> *发件人：*SantoshPK <[email-masked]>
> *收件人：*Greg Mirsky <[email-masked]>;
> *抄送人：[email-masked] 
> <[email-masked]>;Dinesh Dutt <[email-masked]>;rtg-bfd 
> WG <[email-masked]>;Joel M. Halpern <[email-masked]>;T. Sridhar 
> <[email-masked]>;[email-masked] <[email-masked]>;
> *日 期 ：*2019年09月23日 05:39
> *主 题 ：**Re: BFD over VXLAN: Trapping BFD Control packet at VTEP*
> Greg,
> On Fri, Aug 16, 2019 at 4:59 AM Greg Mirsky <gregimirsky@gmail...com 
> <mailto:[email-masked]>> wrote:
> 
>     Hi Santosh,
>     thank you for your comments. Please find my notes in-lined and
>     tagged GIM>>.
> 
>     Regards,
>     Greg
> 
>     On Tue, Aug 13, 2019 at 10:24 PM Santosh P K
>     <[email-masked] <mailto:[email-masked]>>
>     wrote:
> 
>         Greg,
>             Thanks for updated version of document. Here are few
>         comments on new draft.
> 
>         Section 4:
>         I think we may need to give background on why we are saying MAC
>         address MUST not be one of tenant's MAC address. Like in this
>         thread we have discussed one of the tenant could have borrowed
>         the same VTEP mac address and we if we have to use BFD then we
>         need to avoid that conflict to ensure BFD packets get observed
>         in the VTEP itself. Should we add a section before 4 to set that
>         context so that above text makes more sense in that context?
> 
> "In some scenarios tenant MAC is borrowed from VTEP MAC address. VXLAN 
> BFD MUST terminate BFD session at VTEP and MUST not forward BFD packets 
> to tenants. To terminate VXLAN BFD packets at VTEP, deployment MUST 
> ensure that there are no tenant VM which barrows VTEP MAC address."
> 
> 
> 
>             IP header:
>         I think we have added some text to address Sridhar comments on
>         why TTL MUST be 1 and dest IP address MUST be 127/8 range. I see
>         that text is missing now.
> 
>         Just with 3 tuple we will not be able to demux packet. We need
>         to consider VNI as well if we have multiple BFD session between
>         same pair of VTEP.
> 
>     GIM>> This is one of comments from Carlos we need to address. Your
>     comment have helped me to form the question:
> 
> [SPK] The goal of the multiple BFD session is to ensure check liveliness 
> of VXLAN tunnel. There is already a good amount of debate on this topic 
> that do we really need that? As per RFC 5884 we are running BFD per LSP 
> and we might hit scale issues there too. I think it is up to operator to 
> decide how they want to use multiple BFD session per VXLAN tunnel. It 
> could be possible that BFD session with special VNI is run at aggressive 
> interval where as MAY have multiple BFD sessions for different VNI at a 
> sedate interval, for that matter they could be running in demand mode as 
> well (run P/F sequence only when there is no data following for that 
> VNI). As WG if we think running multiple BFD session make sense then we 
> might need to add appropriate text.
> 
>     If the goal is to monitor per VNI, then the following text should
>     describe the demultiplexing of the initial BFD Control packet:
> VTEPs for multiple VNI makes sense only if as WG we think that could be 
> use case.
> 
>         On Fri, Aug 9, 2019 at 4:27 AM Greg Mirsky
>         <[email-masked] <mailto:[email-masked]>> wrote:
> 
>             On Wed, Aug 7, 2019 at 2:25 PM Dinesh Dutt <[email-masked]
>             <mailto:[email-masked]>> wrote:
> 
>                 On Wed, Aug 7, 2019 at 9:25 AM Greg Mirsky
>                 <[email-masked] <mailto:[email-masked]>>
>                 wrote:
> 
>                     much appreciate the help you've given me sharing
>                     your expertise. I hope that the updates you will
>                     find in the attached diff and the working copy of
>                     the draft be closer to the acceptable solution for
>                     VTEP-VTEP BFD. Please note, that I'll shortly start
>                     a new discussion thread to address one of Carlos's
>                     questions on the ambiguity of the text on multiple
>                     concurrent sessions between the same pair of VTEPs.
>                     Please review the changes to Sections 4 and 6 and
>                     share your feedback, suggestions, and questions.
> 
>                     Regards,
>                     Greg
> 
>                     On Mon, Aug 5, 2019 at 6:03 PM Dinesh Dutt
>                     <[email-masked] <mailto:[email-masked]>> wrote:
> 
> 
> 
>                         On Mon, Aug 5, 2019 at 5:56 PM Greg Mirsky
>                         <[email-masked]
>                         <mailto:[email-masked]>> wrote:
> 
>                             Hi Dinesh,
>                             I believe that the ability to run BFD
>                             session up to a tenant (VTEP-VTEP-tenant or
>                             tenant-tenant) was never in jeopardy from
>                             this specification.
>                             I'm trying to provide precise specification
>                             on what can be used ad the destination MAC
>                             and IP addresses in the inner frame/packet
>                             as I believe that likely will help to avoid
>                             interoperability issues.
>                             I'm interested to learn some more about the
>                             "martian checking" function. As you know,
>                             martian addresses have been used as
>                             destination IP address in LSP Ping and BFD
>                             over MPLS LSP and PW. I haven't heard that
>                             any silicon feature caused problems for
>                             operators using these tools.
> 
> 
>                         Interesting. I didn't know this aspect of use
>                         with MPLS ping. Did those packets ever go
>                         through a firewall though? In any case, maybe
>                         suggest the use of those addresses with a
>                         statement that this is how LSP does it, but that
>                         other MAC/IP pairs are possible as long as the
>                         conditions of the endpoint owning the MAC/IP was
>                         honored.
> 
>                         Dinesh
> 
> 
>                             Regards,
>                             Greg
> 
>                             On Mon, Aug 5, 2019 at 3:59 PM Dinesh Dutt
>                             <[email-masked] <mailto:[email-masked]>>
>                             wrote:
> 
>                                 Hi Greg,
> 
>                                 That we agree on the problem definition
>                                 is the first step forward. Your original
>                                 document had my cases covered and so I
>                                 was surprised by the track this thread
>                                 took. It doesn't matter, we're back on
>                                 track.
> 
>                                 My recommendation is to not worry about
>                                 specifying the precise MAC/IP address
>                                 used in the inner header. The addresses
>                                 chosen MUST ensure that the packet is
>                                 trapped to the control plane of the VTEP
>                                 and not escape to the tenant if the BFD
>                                 is to the VTEP. Any solution MUST also
>                                 not preclude the use of the BFD by
>                                 tenant systems for that VNI. There are
>                                 many ways an implementer can choose to
>                                 implement this. For example, the inner
>                                 MAC address is whatever the VTEP
>                                 implementer would return if ARP'd for
>                                 the IP address used in the inner header
>                                 in the given VNI. The implementer can
>                                 pick a fixed MAC address, one that they
>                                 own etc. Multiple BFD sessions can be
>                                 run for testing path connectivity on
>                                 more than one VNIs. Limits should be in
>                                 place to avoid overwhelming the receiver
>                                 with BFD messages (you had words about
>                                 this in your currently published
>                                 draft).  If the VNI is irrelevant in the
>                                 test i.e. only the VXLAN pipe at the
>                                 VTEP is being tested. the user can use
>                                 any VNI active on the VTEP on which the
>                                 VTEP owns an IP address.
> 
>                                 I'm concerned about the use of 127/8
>                                 address only because of firewalls or
>                                 implementations that drop packets with
>                                 these addresses as either the source or
>                                 destination. For example, on many
>                                 merchant silicon, I don't believe you
>                                 can turn off martian checking and drops
>                                 *only* for VXLAN-encapsulated BFD
>                                 packets. I don't know what the Linux
>                                 kernel does today on such packets, for
>                                 example (or Hyper-V). I'd like a
>                                 solution that doesn't demand additional
>                                 or new chip functionality or require
>                                 additional middle-box hole punch.
> 
>                                 Why do you feel you MUST to specify the
>                                 MAC/IP address on the inner packet? What
>                                 am I missing here?
> 
>                                 Dinesh
> 
>                                 On Mon, Aug 5, 2019 at 3:04 PM Greg
>                                 Mirsky <[email-masked]
>                                 <mailto:[email-masked]>> wrote:
> 
>                                     Hi Dinesh,
>                                     what do you see as the way forward?
>                                     I agree, that the proposed text
>                                     doesn't work for multi-VNI
>                                     concurrent monitoring because these
>                                     VNIs are tenant's VNIs. And in that
>                                     case, we need to specify another
>                                     mechanism to trap the BFD Control
>                                     packet at VTEP. It seems that VTEP's
>                                     Ethernet address must be used as the
>                                     destination MAC address in the inner
>                                     Ethernet frame. The destination IP
>                                     address may be either VTEP's address
>                                     of martian (I do prefer martian).
>                                     Let me give it  try:
>                                     NEW TEXT:
> 
>                                         To monitor continuity of the
>                                         path between two VTEPs, an
>                                         operator MUST select a VNI
>                                         number to be used as Management
>                                         VNI. Management VNI number MUST
>                                         NOT be one of the tenant's VNIs
>                                         to prevent sending VXLAN packets
>                                         received on Management VNI to a
>                                         tenant. VNI number 1 is
>                                         RECOMMENDED as the default for
>                                         Management VNI. [Ed.note: What
>                                         we set the Destination MAC to?
>                                         Can it be invalid MAC that MUST
>                                         be ignored on receipt?]
> 
>                                         If an implementation supports
>                                         concurrent monitoring of
>                                         multiple VNIs, then the value of
>                                         VNI number MAY be one of
>                                         tenant's VNIs. The destination
>                                         MAC address in the inner
>                                         Ethernet frame encapsulating BFD
>                                         The destination IP address of
>                                         the inner IP packet MUST be
>                                         selected from the range 127/8
>                                         for IPv4, and for IPv6 from the
>                                         range 0:0:0:0:0:FFFF:7F00:0/104.
>                                         The TTL value in the inner IP
>                                         header MUST be set to 1.
> 
>                                     Regards,
>                                     Greg
> 
>                                     On Sun, Aug 4, 2019 at 9:07 AM
>                                     Dinesh Dutt <[email-masked]
>                                     <mailto:[email-masked]>> wrote:
> 
>                                         Hi Greg,
> 
>                                         why you're running BFD over
>                                         VXLAN between VTEPs. I wasn't
>                                         arguing against it at all. All I
>                                         was saying was pointing to the
>                                         limitations of the use of
>                                         management VNI. I spoke to some
>                                         operators who're running EVPN
>                                         and mentioned the discussion on
>                                         this thread. They concur that
>                                         they're using specific VNIs to
>                                         test connectivity over that VNI
>                                         between VTEPs to ensure
>                                         misconfiguration doesn't lead to
>                                         blackholes. My statements are
>                                         based in real world operator
>                                         experience. And I was providing
>                                         language that ensured packets
>                                         didn't leak across to tenants
>                                         when they were destined to VTEPs.
> 
>                                         Dinesh
> 
>                                         On Sat, Aug 3, 2019 at 10:34 AM
>                                         Greg Mirsky
>                                         <[email-masked]
>                                             implementations process
>                                             VXLAN header and the inner
>                                             Ethernet frame. These are
>                                             very helpful in achieving
>                                             the workable solution for
>                                             the problem at hand.
>                                             You've noted that a path
>                                             between VTEPs may be
>                                             monitored in the underlay
>                                             network by merely
>                                             establishing a BFD session.
>                                             That is true, but by using
>                                             BFD with VXLAN encapsulation
>                                             between the pair of VTEPs we
>                                             are extending the OAM domain
>                                             by including, to some
>                                             extent, VXLAN forwarding
>                                             engine. Abstract in RFC 5880
>                                             defines the goal and the
>                                             domain in which BFD protocol
>                                             can detect a fault as:
>                                                 This document describes
>                                             a protocol intended to
>                                             detect faults in the
>                                                 bidirectional path
>                                             between two forwarding
>                                             engines, including
>                                                 interfaces, data
>                                             link(s), and to the extent
>                                             possible the forwarding
>                                                 engines themselves, with
>                                             Thus, BFD in the underlay
>                                             will exercise a part of IP
>                                             forwarding engine while BFD
>                                             with VXLAN encapsulation,
>                                             ran between the same pair of
>                                             VTEPs, extends the OAM
>                                             domain. At the same time,
>                                             defining BFD between tenant
>                                             systems in outside the goal
>                                             of this specification. But
>                                             VXLAN BFD session between
>                                             VTEPs may be useful in
>                                             monitoring e2e path between
>                                             tenants, as described in the
>                                             update to -07:
>                                                 At the same time, a
>                                             service layer BFD session
>                                             may be used between the
>                                                 tenants of VTEPs IP1 and
>                                             IP2 to provide end-to-end
>                                             fault management.
>                                                 In such case, for VTEPs
>                                             BFD control packets of that
>                                             session are
>                                                 indistinguishable from
>                                             data packets.  If end-to-end
>                                             defect detection
>                                                 is realized as the set
>                                             of concatenated OAM domains,
>                                             e.g., VM1-1 - IP1
>                                                 -- IP2 - VM2-1, then the
>                                             BFD session over VXLAN
>                                             between VTEPs SHOULD
>                                                 follow the procedures
>                                             described in Section 6.8.17
>                                             [RFC5880].
>                                             On Fri, Aug 2, 2019 at 5:43
>                                             PM Dinesh Dutt
>                                             <[email-masked]
>                                                 What I mean is "How do
>                                                 you infer that it
>                                                 excludes the case I'm
>                                                 On Fri, Aug 2, 2019 at
>                                                 5:41 PM Dinesh Dutt
>                                                     On Fri, Aug 2, 2019
>                                                     at 5:38 PM Joel M.
>                                                         I am going by
>                                                         what the draft
>                                                         says its purpose
>                                                         is.  If you
>                                                         the draft to
>                                                         fulfill a
>                                                         different
>                                                         purpose, then
>                                                         either ask the
>                                                         take this draft
>                                                         back to the WG,
>                                                         or write a
>                                                         separate draft.
>                                                         As currently
>                                                         written, the
>                                                         behavior Greg
>                                                         proposed meets
>                                                         the needs, and
>                                                         does so in a way
>                                                         consistent with
>                                                         VxLAN.
>                                                         On 8/2/2019 8:30
>                                                         PM, Dinesh Dutt
>                                                          > What is the
>                                                         stated purpose
>                                                         of this BFD
>                                                         session? The
>                                                         VTEP
>                                                          > determined by
>                                                         the underlay, I
>                                                         don't need
>                                                         VXLAN-encaped
>                                                          > If I want to
>                                                         test the VXLAN
>                                                         encap/decap
>                                                         functionality
>                                                          > single VNI
>                                                         maybe fine. But
>                                                         is this all any
>                                                         network operator
>                                                          > In what
>                                                         situations has
>                                                         this been a
>                                                         problem? I
>                                                         suspect
>                                                          > want to
>                                                         verify path
>                                                         continuity over
>                                                         a specific VNI.
>                                                         If you say this is
>                                                          > not defined
>                                                         by the document,
>                                                         I disagree
>                                                         because the
>                                                         current version
>                                                          > talks about
>                                                         controlling the
>                                                         number of BFD
>                                                         sessions between
>                                                          > (see section
>                                                         3). More
>                                                         importantly,
>                                                         this is a real
>                                                         problem that
>                                                         operators
>                                                          > On Fri, Aug
>                                                         2, 2019 at 5:08
>                                                         PM Joel M.
>                                                         <mailto:[email-masked]>
> 
>                                                          >
>                                                         <mailto:[email-masked]>>>
>                                                         wrote:
>                                                          >
>                                                          >     What is
>                                                         special about
>                                                         the management
>                                                         VNI is precisely
>                                                         that it is NOT a
>                                                          >     tenant
>                                                         VNI.  The VxLAN
>                                                         administration
>                                                         does know how it
>                                                         allocates VNI to
>                                                          >     tenants,
>                                                         and which VNI it
>                                                         has allocated. 
>                                                         In contrast, it
>                                                         does not know
>                                                          >     which IP
>                                                         addresses or MAC
>                                                         adddresses teh
>                                                         tenant is using
>                                                         or may plan
>                                                          >     to use.
>                                                          >
>                                                          >     Yours,
>                                                          >     Joel
>                                                          >
>                                                          >     On
>                                                         8/2/2019 6:41
>                                                         PM, Dinesh Dutt
>                                                         wrote:
>                                                          >      > The
>                                                         assumption of an
>                                                         IP address
>                                                         within any VNI
>                                                         is suspect that way.
>                                                          >      > What's
>                                                         special about a
>                                                         single VNI, the
>                                                         management VNI?
>                                                         The VTEP IP
>                                                          >      >
>                                                         address does not
>                                                         belong in
>                                                         reality in any
>                                                         VNI...
>                                                          >      >
>                                                          >      > Dinesh
>                                                          >      >
>                                                          >      > On
>                                                         Fri, Aug 2, 2019
>                                                         at 3:17 PM Joel
>                                                         M. Halpern
>                                                          >   
>                                                           <[email-masked] <mailto:[email-masked]> <mailto:[email-masked] <mailto:[email-masked]>>
>                                                          >      >
>                                                         <mailto:[email-masked]>>>>
>                                                         wrote:
>                                                          >      >
>                                                          >      >   
>                                                           Your response
>                                                         seems to miss
>                                                         two points:
>                                                          >      >
>                                                          >      >   
>                                                           First, the
>                                                         problem you
>                                                         describe is not
>                                                         what the
>                                                         document says
>                                                          >     it is
>                                                          >      >   
>                                                           solving.  To
>                                                         the degree it
>                                                         discusses it at
>                                                         all, the document
>                                                          >     says "
>                                                          >      >       In
>                                                          >      >   
>                                                           most cases, a
>                                                         single BFD
>                                                         session is
>                                                         sufficient for
>                                                         the given
>                                                          >     VTEP to
>                                                          >      >   
>                                                           monitor the
>                                                         reachability of
>                                                         a remote VTEP,
>                                                         regardless of the
>                                                          >     number of
>                                                          >      >   
>                                                           VNIs in common. "
>                                                          >      >
>                                                          >      >   
>                                                           Second, you
>                                                         assume the
>                                                         existence of an
>                                                         IP address for a
>                                                         VTEP
>                                                          >     within a
>                                                          >      >   
>                                                           VNI.  As with
>                                                         the MAC address,
>                                                         the VTEP does
>                                                         not have an IP
>                                                          >     address
>                                                          >      >   
>                                                           within the
>                                                         VNI.  Some
>                                                         implementations
>                                                         may have created
>                                                         such a
>                                                          >     thing,
>                                                          >      >     but
>                                                          >      >   
>                                                           the general
>                                                         construct, as
>                                                         defined to date,
>                                                         does not support
>                                                         such.
>                                                          >      >
>                                                          >      >     In
>                                                         short, you are
>                                                         requiring a
>                                                         behavior that
>                                                         violates the
>                                                          >     architectural
>                                                          >      >   
>                                                           structure of
>                                                         overlay /
>                                                         underlay
>                                                         separation, and
>                                                         common
>                                                          >     usage. 
>                                                         And you
>                                                          >      >   
>                                                           are doing so
>                                                         to support a use
>                                                         case that the
>                                                         working group
>                                                         has not
>                                                          >      >   
>                                                           indicated in
>                                                         the document as
>                                                         important.
>                                                          >      >
>                                                          >      >     Yours,
>                                                          >      >     Joel
>                                                          >      >
>                                                          >      >     On
>                                                         8/2/2019 5:01
>                                                         PM, Dinesh Dutt
>                                                         wrote:
>                                                          >      >      >
>                                                         Joel,
>                                                          >      >      >
>                                                          >      >      >
>                                                         You understood
>                                                         correctly.
>                                                          >      >      >
>                                                          >      >      >
>                                                         The VNIs may not
>                                                         share fate due
>                                                         to
>                                                         misconfiguration. And
>                                                         I
>                                                          >     strongly
>                                                          >      >      >
>                                                         suspect someone
>                                                         will want to use
>                                                         BFD for that
>                                                         because its
>                                                          >     about
>                                                          >      >   
>                                                           checking
>                                                          >      >      >
>                                                         path continuity
>                                                         as stated by the
>                                                         draft. As long
>                                                         as there's a
>                                                          >      >   
>                                                           valid IP
>                                                          >      >      >
>                                                         (because it's
>                                                         BFD) owned by
>                                                         the VTEP in that
>                                                         VNI, you can
>                                                          >     use BFD in
>                                                          >      >      >
>                                                         that VNI. Thats
>                                                         all that you
>                                                         need to
>                                                         dictate.  That
>                                                         IP address
>                                                          >      >   
>                                                           has a MAC
>                                                          >      >      >
>                                                         address and you
>                                                         can use that on
>                                                         the inner frame.
>                                                         That is
>                                                          >     all normal
>                                                          >      >      >
>                                                         VXLAN
>                                                         processing. The
>                                                         outer IP is
>                                                         always that of
>                                                         the VTEP.
>                                                          >      >      >
>                                                          >      >      >
>                                                         Dinesh
>                                                          >      >      >
>                                                          >      >      >
>                                                         On Fri, Aug 2,
>                                                         2019 at 11:03 AM
>                                                         Joel M. Halpern
>                                                          >      >   
>                                                           <[email-masked] <mailto:[email-masked]> <mailto:[email-masked] <mailto:[email-masked]>>
>                                                          >   
>                                                           <mailto:[email-masked] <mailto:[email-masked]> <mailto:[email-masked] <mailto:[email-masked]>>>
>                                                          >      >      >
>                                                           <mailto:[email-masked] <mailto:[email-masked]> <mailto:[email-masked] <mailto:[email-masked]>>>>> wrote:
>                                                          >      >      >
>                                                          >      >     
>                                                          >     If I am
>                                                         reading your
>                                                         various emails
>                                                         correctly Dinesh
>                                                          >     (and I
>                                                          >      >   
>                                                           may have
>                                                          >      >     
>                                                          >     missed
>                                                         something) you
>                                                         are trying to
>                                                         use the MAC address
>                                                          >      >   
>                                                           because you
>                                                          >      >     
>                                                          >     want
>                                                          >      >     
>                                                          >     to be
>                                                         able to send
>                                                         these BFD
>                                                         packets over
>                                                         arbitrary VNI to
>                                                          >      >   
>                                                           monitor the
>                                                          >      >     
>                                                          >     VNI. 
>                                                         That is not a
>                                                         requirement
>                                                         identified in the
>                                                          >     document.
>                                                          >      >     It
>                                                         is not
>                                                          >      >     
>                                                          >     even a
>                                                         problem I
>                                                         understand,
>                                                         since all the
>                                                         VNI between an
>                                                          >      >   
>                                                           ingress and
>                                                          >      >     
>                                                          >     egress
>                                                         VTEP share fate.
>                                                          >      >      >
>                                                          >      >     
>                                                          >     Yours,
>                                                          >      >     
>                                                          >     Joel
>                                                          >      >      >
>                                                          >      >     
>                                                          >     On
>                                                         8/2/2019 1:44
>                                                         PM, Dinesh Dutt
>                                                         wrote:
>                                                          >      >     
>                                                          >      > Thanks
>                                                         for verifying
>                                                         this. On Linux
>                                                         and hardware
>                                                          >     routers
>                                                          >      >   
>                                                           that I'm
>                                                          >      >     
>                                                          >     aware
>                                                          >      >     
>                                                          >      > of
>                                                         (Cisco circa
>                                                         2012 and
>                                                         Cumulus), the
>                                                         physical MAC
>                                                          >     address is
>                                                          >      >     
>                                                          >     reused
>                                                          >      >     
>                                                          >      > across
>                                                         the VNIs on the
>                                                         VTEP. Did you
>                                                         check on a non-VMW
>                                                          >      >   
>                                                           device?
>                                                          >      >     
>                                                          >     This is
>                                                          >      >     
>                                                          >      > more
>                                                         for my own
>                                                         curiosity.
>                                                          >      >     
>                                                          >      >
>                                                          >      >     
>                                                          >      > To
>                                                         address the
>                                                         general case,
>                                                         can we not define a
>                                                          >      >   
>                                                           well-known (or
>                                                          >      >     
>                                                          >     reserve
>                                                          >      >     
>                                                          >      > one)
>                                                         unicast MAC
>                                                         address for use
>                                                         with VTEP? If
>                                                         the MAC
>                                                          >      >   
>                                                           address is
>                                                          >      >     
>                                                          >      >
>                                                         configurable in
>                                                         BFD command,
>                                                         this can be moot.
>                                                          >      >     
>                                                          >      >
>                                                          >      >     
>                                                          >      > Dinesh
>                                                          >      >     
>                                                          >      >
>                                                          >      >     
>                                                          >      > On
>                                                         Fri, Aug 2, 2019
>                                                         at 10:27 AM
>                                                         Santosh P K
>                                                          >      >     
>                                                          >      >
>                                                           <mailto:[email-masked] <mailto:[email-masked]>>
>                                                          >      >   
>                                                           <mailto:santosh.pallagatti@gmail...com <mailto:[email-masked]>
>                                                          >   
>                                                           <mailto:[email-masked] <mailto:[email-masked]>>>
>                                                          >      >     
>                                                          >   
>                                                           <mailto:[email-masked] <mailto:[email-masked]>
>                                                          >   
>                                                           <mailto:[email-masked] <mailto:[email-masked]>>
>                                                          >      >   
>                                                           <mailto:santosh.pallagatti@gmail...com <mailto:[email-masked]>
>                                                          >   
>                                                           <mailto:[email-masked] <mailto:[email-masked]>>>>
>                                                          >      >     
>                                                          >   
>                                                           <mailto:[email-masked] <mailto:[email-masked]>
>                                                          >   
>                                                           <mailto:[email-masked] <mailto:[email-masked]>>
>                                                          >      >   
>                                                           <mailto:santosh.pallagatti@gmail...com <mailto:[email-masked]>
>                                                          >   
>                                                           <mailto:[email-masked] <mailto:[email-masked]>>>
>                                                          >      >     
>                                                          >   
>                                                           <mailto:[email-masked] <mailto:[email-masked]>
>                                                          >   
>                                                           <mailto:[email-masked] <mailto:[email-masked]>>
>                                                          >      >   
>                                                           <mailto:santosh.pallagatti@gmail...com <mailto:[email-masked]>
>                                                          >   
>                                                           <mailto:[email-masked] <mailto:[email-masked]>>>>>> wrote:
>                                                          >      >     
>                                                          >      >
>                                                          >      >     
>                                                          >      >     I
>                                                         have cross
>                                                         checked point
>                                                         raised about MAC
>                                                         address
>                                                          >      >   
>                                                           usage. It is
>                                                          >      >     
>                                                          >      >   
>                                                           possible that
>                                                         tenant could be
>                                                         using physical MAC
>                                                          >      >   
>                                                           address and
>                                                          >      >     
>                                                          >     when a
>                                                          >      >     
>                                                          >      >   
>                                                           packet comes
>                                                         with valid VNI
>                                                         with a MAC address
>                                                          >     that is
>                                                          >      >     being
>                                                          >      >     
>                                                          >     used by
>                                                          >      >     
>                                                          >      >   
>                                                           tenant then
>                                                         packet will be
>                                                         sent to that tenant.
>                                                          >     This rules
>                                                          >      >     
>                                                          >     out the
>                                                          >      >     
>                                                          >      >   
>                                                           fact that we
>                                                         could use
>                                                         physical MAC
>                                                         address as
>                                                          >     inner
>                                                          >      >     MAC to
>                                                          >      >     
>                                                          >     ensure
>                                                          >      >     
>                                                          >      >   
>                                                           packets get
>                                                         terminated at
>                                                         VTEP itself.
>                                                          >      >     
>                                                          >      >
>                                                          >      >     
>                                                          >      >     Thanks
>                                                          >      >     
>                                                          >      >   
>                                                           Santosh P K
>                                                          >      >     
>                                                          >      >
>                                                          >      >     
>                                                          >      >     On
>                                                         Wed, Jul 31,
>                                                         2019 at 11:00 AM
>                                                         Santosh P K
>                                                          >      >     
>                                                          >      >   
>                                                           <[email-masked] <mailto:[email-masked]>
>                                                          >   
>                                                           <mailto:[email-masked] <mailto:[email-masked]>>
>                                                          >      >   
>                                                           <mailto:santosh.pallagatti@gmail...com <mailto:[email-masked]>
>                                                          >   
>                                                           <mailto:[email-masked] <mailto:[email-masked]>>>
>                                                          >      >     
>                                                          >   
>                                                           <mailto:[email-masked] <mailto:[email-masked]>
>                                                          >   
>                                                           <mailto:[email-masked] <mailto:[email-masked]>>
>                                                          >      >   
>                                                           <mailto:santosh.pallagatti@gmail...com <mailto:[email-masked]>
>                                                          >   
>                                                           <mailto:[email-masked] <mailto:[email-masked]>>>>
>                                                          >      >     
>                                                          >   
>                                                           <mailto:[email-masked] <mailto:[email-masked]>
>                                                          >   
>                                                           <mailto:[email-masked] <mailto:[email-masked]>>
>                                                          >      >   
>                                                           <mailto:santosh.pallagatti@gmail...com <mailto:[email-masked]>
>                                                          >   
>                                                           <mailto:[email-masked] <mailto:[email-masked]>>>
>                                                          >      >     
>                                                          >   
>                                                           <mailto:[email-masked] <mailto:[email-masked]>
>                                                          >   
>                                                           <mailto:[email-masked] <mailto:[email-masked]>>
>                                                          >      >   
>                                                           <mailto:santosh.pallagatti@gmail...com <mailto:[email-masked]>
>                                                          >   
>                                                           <mailto:[email-masked] <mailto:[email-masked]>>>>>>
>                                                          >      >     
>                                                          >      >     wrote:
>                                                          >      >     
>                                                          >      >
>                                                          >      >     
>                                                          >      >       
>                                                           Joel,
>                                                          >      >     
>                                                          >      >       
>                                                               Thanks for
>                                                         your inputs. I
>                                                         checked
>                                                          >      >   
>                                                           implementation
>                                                         within
>                                                          >      >     
>                                                          >      >       
>                                                           Vmware.
>                                                         Perhaps I should
>                                                         have been more clear
>                                                          >      >   
>                                                           about MAC
>                                                          >      >     
>                                                          >     address
>                                                          >      >     
>                                                          >      >       
>                                                           space while
>                                                         checking
>                                                         internally. I
>                                                         will cross
>                                                          >      >   
>                                                           check again for
>                                                          >      >     
>                                                          >      >       
>                                                           the same and
>                                                         get back on this
>                                                         list.
>                                                          >      >     
>                                                          >      >
>                                                          >      >     
>                                                          >      >       
>                                                           Thanks
>                                                          >      >     
>                                                          >      >       
>                                                           Santosh P K
>                                                          >      >     
>                                                          >      >
>                                                          >      >     
>                                                          >      >       
>                                                           On Wed, Jul
>                                                         31, 2019 at
>                                                         10:54 AM Joel M.
>                                                          >     Halpern
>                                                          >      >     
>                                                          >      >       
>                                                           <[email-masked] <mailto:jmh@joelhalpern...com>
>                                                          >   
>                                                           <mailto:[email-masked] <mailto:[email-masked]>> <mailto:[email-masked] <mailto:[email-masked]>
>                                                          >   
>                                                          >      >   
>                                                           <mailto:[email-masked] <mailto:[email-masked]> <mailto:[email-masked] <mailto:[email-masked]>>
>                                                          >   
>                                                           <mailto:[email-masked] <mailto:[email-masked]> <mailto:[email-masked] <mailto:[email-masked]>>>>
>                                                          >      >     
>                                                          >   
>                                                           <mailto:[email-masked] <mailto:[email-masked]>> <mailto:[email-masked] <mailto:[email-masked]>
>                                                          >   
>                                                          >      >   
>                                                           <mailto:[email-masked] <mailto:[email-masked]> <mailto:[email-masked] <mailto:[email-masked]>>
>                                                          >   
>                                                           <mailto:[email-masked] <mailto:[email-masked]> <mailto:[email-masked] <mailto:[email-masked]>>>>>> wrote:
>                                                          >      >     
>                                                          >      >
>                                                          >      >     
>                                                          >      >       
>                                                               Sorry to
>                                                         ask a stupid
>                                                         question.  Whose
>                                                          >      >   
>                                                           implementation?
>                                                          >      >     
>                                                          >      >
>                                                          >      >     
>                                                          >      >       
>                                                               The reason
>                                                         I ask is that as
>                                                         far as I
>                                                          >     can tell,
>                                                          >      >   
>                                                           since the
>                                                          >      >     
>                                                          >      >       
>                                                               tenant
>                                                         does not
>                                                          >      >     
>                                                          >      >       
>                                                               have any
>                                                         control access
>                                                         to the VTEP,
>                                                          >     there is no
>                                                          >      >     
>                                                          >     reason for
>                                                          >      >     
>                                                          >      >       
>                                                               the VTEP to
>                                                          >      >     
>                                                          >      >       
>                                                               have a MAC
>                                                         address in the
>                                                         tenant
>                                                          >     space. 
>                                                         Yes, the
>                                                          >      >     
>                                                          >     device has
>                                                          >      >     
>                                                          >      >       
>                                                               a physical
>                                                          >      >     
>                                                          >      >       
>                                                               MAC
>                                                         address.  But
>                                                         the tenant could
>                                                         well be
>                                                          >      >   
>                                                           using that MAC
>                                                          >      >     
>                                                          >      >       
>                                                               address.  Yes,
>                                                          >      >     
>                                                          >      >       
>                                                               they would
>                                                         be violating the
>                                                         Ethernet spec.
>                                                          >      >   
>                                                           But the whole
>                                                          >      >     
>                                                          >      >       
>                                                               point of
>                                                          >      >     
>                                                          >      >       
>                                                             
>                                                           segregation is
>                                                         not to care
>                                                         about such
>                                                          >     issues.
>                                                          >      >     
>                                                          >      >
>                                                          >      >     
>                                                          >      >       
>                                                               On the
>                                                         other hand, if
>                                                         you tell me that
>                                                          >     the VMWare
>                                                          >      >     
>                                                          >      >       
>                                                             
>                                                           implementation
>                                                         has an
>                                                          >      >     
>                                                          >      >       
>                                                               Ethernet
>                                                         address that is
>                                                         part of the tenant
>                                                          >      >   
>                                                           space, well,
>                                                          >      >     
>                                                          >      >       
>                                                               they made up
>                                                          >      >     
>                                                          >      >       
>                                                               this
>                                                         particular game.
>                                                          >      >     
>                                                          >      >
>                                                          >      >     
>                                                          >      >       
>                                                               Yours,
>                                                          >      >     
>                                                          >      >       
>                                                               Joel
>                                                          >      >     
>                                                          >      >
>                                                          >      >     
>                                                          >      >       
>                                                               On
>                                                         7/31/2019 1:44
>                                                         PM, Santosh P K
>                                                         wrote:
>                                                          >      >     
>                                                          >      >       
>                                                                > I have
>                                                         checked with
>                                                         implementation
>                                                          >     in data
>                                                          >      >     path.
>                                                          >      >     
>                                                          >     When we
>                                                          >      >     
>                                                          >      >       
>                                                               receive a
>                                                          >      >     
>                                                          >      >       
>                                                                > packet
>                                                         with valid VNI
>                                                         then lookup
>                                                          >     for MAC will
>                                                          >      >     
>                                                          >     happen and
>                                                          >      >     
>                                                          >      >       
>                                                               it is VTEP own
>                                                          >      >     
>                                                          >      >       
>                                                                > MAC
>                                                         then it will be
>                                                         trapped to control
>                                                          >      >   
>                                                           plane for
>                                                          >      >     
>                                                          >      >       
>                                                             
>                                                           processing. I
>                                                         think we
>                                                          >      >     
>                                                          >      >       
>                                                                > can
>                                                         have following
>                                                         options
>                                                          >      >     
>                                                          >      >       
>                                                                > 1.
>                                                         Optional
>                                                         managment VNI
>                                                          >      >     
>                                                          >      >       
>                                                                > 2.
>                                                         Mandatory inner
>                                                         MAC set to VTEP mac
>                                                          >      >     
>                                                          >      >       
>                                                                > 3.
>                                                         Inner IP TTL set
>                                                         to 1 to avoid
>                                                          >      >   
>                                                           forwarding of
>                                                         packet
>                                                          >      >     
>                                                          >      >       
>                                                               via inner IP
>                                                          >      >     
>                                                          >      >       
>                                                                > address.
>                                                          >      >     
>                                                          >      >       
>                                                                >
>                                                          >      >     
>                                                          >      >       
>                                                                >
>                                                          >      >     
>                                                          >      >       
>                                                                > Thoughts?
>                                                          >      >     
>                                                          >      >       
>                                                                >
>                                                          >      >     
>                                                          >      >       
>                                                                > Thansk
>                                                          >      >     
>                                                          >      >       
>                                                                > Santosh P K
>                                                          >      >     
>                                                          >      >       
>                                                                >
>                                                          >      >     
>                                                          >      >       
>                                                                > On Wed,
>                                                         Jul 31, 2019 at
>                                                         9:20 AM Greg
>                                                          >     Mirsky
>                                                          >      >     
>                                                          >      >       
>                                                             
>                                                           <[email-masked] <mailto:[email-masked]>
>                                                          >   
>                                                          >      >   
>                                                           <mailto:[email-masked] <mailto:[email-masked]> <mailto:[email-masked] <mailto:[email-masked]>>>
>                                                          >   
>                                                           <mailto:[email-masked] <mailto:[email-masked]> <mailto:[email-masked] <mailto:[email-masked]>>
>                                                          >      >   
>                                                           <mailto:[email-masked] <mailto:[email-masked]> <mailto:[email-masked] <mailto:[email-masked]>>>>
>                                                          >      >     
>                                                          >   
>                                                           <mailto:[email-masked] <mailto:[email-masked]>> <mailto:[email-masked] <mailto:[email-masked]>
>                                                          >   
>                                                          >      >   
>                                                           <mailto:[email-masked] <mailto:[email-masked]> <mailto:[email-masked] <mailto:[email-masked]>>
>                                                          >   
>                                                           <mailto:[email-masked] <mailto:[email-masked]> <mailto:[email-masked] <mailto:[email-masked]>>>>>
>                                                          >      >     
>                                                          >      >       
>                                                                >
>                                                          >      >   
>                                                           <mailto:[email-masked] <mailto:[email-masked]> <mailto:[email-masked] <mailto:[email-masked]>>>
>                                                          >      >     
>                                                          >   
>                                                           <mailto:[email-masked] <mailto:[email-masked]>> <mailto:[email-masked] <mailto:[email-masked]>
>                                                          >   
>                                                          >      >     
>                                                          >      >       
>                                                          >      >   
>                                                           <mailto:[email-masked] <mailto:[email-masked]> <mailto:[email-masked] <mailto:[email-masked]>>>
>                                                          >      >     
>                                                          >   
>                                                          >      >   
>                                                           <mailto:[email-masked] <mailto:[email-masked]>>>>>>> wrote:
>                                                          >      >     
>                                                          >      >       
>                                                                >
>                                                          >      >     
>                                                          >      >       
>                                                                >     Hi
>                                                         Dinesh,
>                                                          >      >     
>                                                          >      >       
>                                                                >   
>                                                           thank you for
>                                                         your consideration
>                                                          >     of the
>                                                          >      >     
>                                                          >     proposal and
>                                                          >      >     
>                                                          >      >       
>                                                               questions.
>                                                         What
>                                                          >      >     
>                                                          >      >       
>                                                                >   
>                                                           would you see
>                                                         as the scope of
>                                                          >     testing the
>                                                          >      >     
>                                                          >      >       
>                                                             
>                                                           connectivity
>                                                         for the
>                                                          >      >     
>                                                          >      >       
>                                                                >   
>                                                           specific VNI?
>                                                         If it is
>                                                          >      >   
>                                                           tenant-to-tenant, then
>                                                          >      >     
>                                                          >     VTEPs
>                                                          >      >     
>                                                          >      >       
>                                                               will treat
>                                                         these
>                                                          >      >     
>                                                          >      >       
>                                                                >   
>                                                           packets as
>                                                         regular user
>                                                         frames. More
>                                                          >      >   
>                                                           likely, these
>                                                          >      >     
>                                                          >      >       
>                                                               could be
>                                                         Layer 2
>                                                          >      >     
>                                                          >      >       
>                                                                >   
>                                                           OAM, e.g. CCM
>                                                         frames. The reason
>                                                          >     to use
>                                                          >      >   
>                                                           127/8 for
>                                                          >      >     
>                                                          >      >       
>                                                               IPv4, and
>                                                          >      >     
>                                                          >      >       
>                                                                >   
>                                                          >      >     to
>                                                         safeguard
>                                                          >      >     
>                                                          >      >       
>                                                          >      >     
>                                                          >      >       
>                                                                >   
>                                                         frames with BFD
>                                                         Control
>                                                          >      >     to a
>                                                          >      >     
>                                                          >      >     
>                                                          >      >       
>                                                                >   
>                                                           You've
>                                                         suggested using
>                                                         a MAC
>                                                          >      >   
>                                                          >      >     
>                                                          >      >       
>                                                               control
>                                                         packet at
>                                                          >      >     
>                                                          >      >       
>                                                                >   
>                                                           VTEP. What
>                                                         that address
>                                                         could be? We
>                                                          >      >   
>                                                          >      >     
>                                                          >      >       
>                                                          >      >     
>                                                          >      >       
>                                                                >   
>                                                         and VTEP's MAC and
>                                                          >      >     raised
>                                                          >      >     
>                                                          >      >     
>                                                          >      >       
>                                                          >      >     
>                                                          >      >       
>                                                                >   
>                                                          >      >   
>                                                          >      >     
>                                                          >      >     
>                                                          >      >       
>                                                          >      >     
>                                                          >      >       
>                                                                >   
>                                                          >      >     of
>                                                         using
>                                                          >      >     
>                                                          >      >       
>                                                         VLAN.
>                                                          >      >     
>                                                          >      >       
>                                                                >
>                                                          >      >     
>                                                          >      >       
>                                                                >   
>                                                          >      >     
>                                                          >      >       
>                                                                >     Greg
>                                                          >      >     
>                                                          >      >       
>                                                                >
>                                                          >      >     
>                                                          >      >       
>                                                                >     On
>                                                         Wed, Jul 31,
>                                                          >      >     Dutt
>                                                          >      >     
>                                                          >      >       
>                                                           <[email-masked] <mailto:[email-masked]>
>                                                          >   
>                                                           <mailto:[email-masked] <mailto:[email-masked]>> <mailto:[email-masked] <mailto:[email-masked]>
>                                                          >   
>                                                          >      >   
>                                                           <mailto:[email-masked] <mailto:[email-masked]> <mailto:[email-masked] <mailto:[email-masked]>>
>                                                          >   
>                                                           <mailto:[email-masked] <mailto:[email-masked]> <mailto:[email-masked] <mailto:[email-masked]>>>>
>                                                          >      >     
>                                                          >   
>                                                           <mailto:[email-masked] <mailto:[email-masked]> <mailto:[email-masked] <mailto:[email-masked]>>
>                                                          >   
>                                                           <mailto:[email-masked] <mailto:[email-masked]> <mailto:[email-masked] <mailto:[email-masked]>>>
>                                                          >      >   
>                                                           <mailto:[email-masked] <mailto:[email-masked]> <mailto:[email-masked] <mailto:[email-masked]>>
>                                                          >   
>                                                           <mailto:[email-masked] <mailto:[email-masked]> <mailto:[email-masked] <mailto:[email-masked]>>>>>
>                                                          >      >     
>                                                          >      >       
>                                                                >   
>                                                          >      >   
>                                                           <mailto:[email-masked] <mailto:[email-masked]> <mailto:[email-masked] <mailto:[email-masked]>>>
>                                                          >      >     
>                                                          >   
>                                                           <mailto:[email-masked] <mailto:[email-masked]> <mailto:[email-masked] <mailto:[email-masked]>>
>                                                          >   
>                                                           <mailto:[email-masked] <mailto:[email-masked]> <mailto:[email-masked] <mailto:[email-masked]>>>>
>                                                          >      >   
>                                                           <mailto:[email-masked] <mailto:[email-masked]> <mailto:[email-masked] <mailto:[email-masked]>>
>                                                          >   
>                                                           <mailto:[email-masked] <mailto:[email-masked]> <mailto:[email-masked] <mailto:[email-masked]>>>
>                                                          >      >     
>                                                          >   
>                                                           <mailto:[email-masked] <mailto:[email-masked]> <mailto:[email-masked] <mailto:[email-masked]>>
>                                                          >   
>                                                           <mailto:[email-masked] <mailto:[email-masked]> <mailto:[email-masked] <mailto:[email-masked]>>>>>>>
>                                                          >      >     
>                                                          >      >       
>                                                               wrote:
>                                                          >      >     
>                                                          >      >       
>                                                                >
>                                                          >      >     
>                                                          >      >       
>                                                                >       
>                                                           Hi Greg,
>                                                          >      >     
>                                                          >      >       
>                                                                >
>                                                          >      >     
>                                                          >      >       
>                                                                >       
>                                                           As long as the
>                                                         inner MAC
>                                                          >     address is
>                                                          >      >     such
>                                                          >      >     
>                                                          >     that the
>                                                          >      >     
>                                                          >      >       
>                                                               packet is
>                                                          >      >     
>                                                          >      >       
>                                                                >       
>                                                           trapped to the
>                                                         CPU, it should be
>                                                          >      >   
>                                                           fine for
>                                                          >      >     
>                                                          >     use as
>                                                          >      >     
>                                                          >      >       
>                                                               an inner
>                                                         MAC is
>                                                          >      >     
>                                                          >      >       
>                                                                >       
>                                                           it not?
>                                                         Stating that is
>                                                          >     better than
>                                                          >      >   
>                                                           trying to
>                                                          >      >     
>                                                          >      >       
>                                                               force a
>                                                         management
>                                                          >      >     
>                                                          >      >       
>                                                                >       
>                                                           VNI. What if
>                                                         someone wants
>                                                          >     to test
>                                                          >      >     
>                                                          >     connectivity
>                                                          >      >     
>                                                          >      >       
>                                                               on a specific
>                                                          >      >     
>                                                          >      >       
>                                                                >       
>                                                           VNI? I would
>                                                         not pick a
>                                                          >     loopback IP
>                                                          >      >     
>                                                          >     address for
>                                                          >      >     
>                                                          >      >       
>                                                               this since
>                                                         that
>                                                          >      >     
>                                                          >      >       
>                                                                >       
>                                                           address range
>                                                         is host/node local
>                                                          >      >   
>                                                           only. Is
>                                                          >      >     
>                                                          >     there a
>                                                          >      >     
>                                                          >      >       
>                                                               reason you're
>                                                          >      >     
>                                                          >      >       
>                                                                >       
>                                                           not using the
>                                                         VTEP IP as the
>                                                          >     inner IP
>                                                          >      >     
>                                                          >     address ?
>                                                          >      >     
>                                                          >      >       
>                                                                >
>                                                          >      >     
>                                                          >      >       
>                                                                >       
>                                                           Dinesh
>                                                          >      >     
>                                                          >      >       
>                                                                >
>                                                          >      >     
>                                                          >      >       
>                                                                >       
>                                                           On Wed, Jul
>                                                         31, 2019 at 5:48 AM
>                                                          >      >   
>                                                           Greg Mirsky
>                                                          >      >     
>                                                          >      >       
>                                                                >       
>                                                           <[email-masked] <mailto:[email-masked]>
>                                                          >   
>                                                          >      >   
>                                                           <mailto:[email-masked] <mailto:[email-masked]> <mailto:[email-masked] <mailto:[email-masked]>>>
>                                                          >      >     
>                                                          >   
>                                                           <mailto:[email-masked] <mailto:[email-masked]>> <mailto:[email-masked] <mailto:[email-masked]>
>                                                          >   
>                                                          >      >     
>                                                          >      >       
>                                                          >      >   
>                                                           <mailto:[email-masked] <mailto:[email-masked]> <mailto:[email-masked] <mailto:[email-masked]>>>
>                                                          >      >     
>                                                          >   
>                                                          >      >   
>                                                           <mailto:[email-masked] <mailto:[email-masked]>>>>> <mailto:[email-masked] <mailto:[email-masked]>
>                                                          >   
>                                                          >      >   
>                                                           <mailto:[email-masked] <mailto:[email-masked]> <mailto:[email-masked] <mailto:[email-masked]>>>
>                                                          >      >     
>                                                          >   
>                                                           <mailto:[email-masked] <mailto:[email-masked]>> <mailto:[email-masked] <mailto:[email-masked]>
>                                                          >   
>                                                          >      >     
>                                                          >      >       
>                                                          >      >   
>                                                           <mailto:[email-masked] <mailto:[email-masked]> <mailto:[email-masked] <mailto:[email-masked]>>>
>                                                          >      >     
>                                                          >   
>                                                          >      >   
>                                                           <mailto:[email-masked] <mailto:[email-masked]>>>>>>> wrote:
>                                                          >      >     
>                                                          >      >       
>                                                                >
>                                                          >      >     
>                                                          >      >       
>                                                                >       
>                                                               Dear All,
>                                                          >      >     
>                                                          >      >       
>                                                                >       
>                                                               thank you
>                                                         for your comments,
>                                                          >      >     
>                                                          >   
>                                                           suggestions on
>                                                          >      >     
>                                                          >      >       
>                                                               this issue,
>                                                          >      >     
>                                                          >      >       
>                                                                >       
>                                                               probably
>                                                         the most
>                                                          >     challenging
>                                                          >      >   
>                                                           for this
>                                                          >      >     
>                                                          >      >       
>                                                             
>                                                           specification.
>                                                         In the
>                                                          >      >     
>                                                          >      >       
>                                                                >       
>                                                               course of
>                                                         our discussions,
>                                                          >      >   
>                                                           we've agreed to
>                                                          >      >     
>                                                          >      >       
>                                                               abandon the
>                                                          >      >     
>                                                          >      >       
>                                                                >       
>                                                               request to
>                                                         allocate the
>                                                          >      >   
>                                                           dedicated MAC
>                                                          >      >     
>                                                          >     address
>                                                          >      >     
>                                                          >      >       
>                                                               to be used as
>                                                          >      >     
>                                                          >      >       
>                                                                >       
>                                                               the
>                                                         destination MAC
>                                                          >     address in
>                                                          >      >   
>                                                           the inner
>                                                          >      >     
>                                                          >      >       
>                                                               Ethernet
>                                                         frame.
>                                                          >      >     
>                                                          >      >       
>                                                                >       
>                                                               Also,
>                                                         earlier using VNI
>                                                          >     0 was
>                                                          >      >   
>                                                           changed from
>                                                          >      >     
>                                                          >      >       
>                                                               mandatory
>                                                         to one
>                                                          >      >     
>                                                          >      >       
>                                                                >       
>                                                               of the
>                                                         options an
>                                                          >      >   
>                                                           implementation may
>                                                          >      >     
>                                                          >     offer to
>                                                          >      >     
>                                                          >      >       
>                                                               an operator.
>                                                          >      >     
>                                                          >      >       
>                                                                >       
>                                                               The most
>                                                         recent
>                                                          >   
>                                                           discussion was
>                                                          >      >   
>                                                           whether
>                                                          >      >     
>                                                          >     VTEP's
>                                                          >      >     
>                                                          >      >       
>                                                               MAC address
>                                                          >      >     
>                                                          >      >       
>                                                                >       
>                                                               might be
>                                                         used as the
>                                                          >      >   
>                                                           destination MAC
>                                                          >      >     
>                                                          >     address
>                                                          >      >     
>                                                          >      >       
>                                                               in the inner
>                                                          >      >     
>                                                          >      >       
>                                                                >       
>                                                               Ethernet
>                                                         frame. As I
>                                                          >     recall
>                                                         it, the
>                                                          >      >     
>                                                          >     comments
>                                                          >      >     
>                                                          >      >       
>                                                               from VXLAN
>                                                          >      >     
>                                                          >      >       
>                                                                >       
>                                                               experts
>                                                         equally split
>                                                          >     with one
>                                                          >      >     for it
>                                                          >      >     
>                                                          >     and one
>                                                          >      >     
>                                                          >      >       
>                                                               against. Hence
>                                                          >      >     
>                                                          >      >       
>                                                                >       
>                                                               I would
>                                                         like to propose
>                                                          >     a new
>                                                          >      >   
>                                                           text to
>                                                          >      >     
>                                                          >     resolve
>                                                          >      >     
>                                                          >      >       
>                                                               the issue. The
>                                                          >      >     
>                                                          >      >       
>                                                                >       
>                                                               idea is to
>                                                         let an
>                                                          >     operator
>                                                         select
>                                                          >      >     
>                                                          >     Management
>                                                          >      >     
>                                                          >      >       
>                                                               VNI and use
>                                                          >      >     
>                                                          >      >       
>                                                                >       
>                                                               that VNI
>                                                         in VXLAN
>                                                          >     encapsulation
>                                                          >      >     of BFD
>                                                          >      >     
>                                                          >      >       
>                                                               Control
>                                                         packets:
>                                                          >      >     
>                                                          >      >       
>                                                                >       
>                                                               NEW TEXT:
>                                                          >      >     
>                                                          >      >       
>                                                                >
>                                                          >      >     
>                                                          >      >       
>                                                                >       
>                                                                   An
>                                                         operator MUST
>                                                          >     select a VNI
>                                                          >      >     
>                                                          >     number to
>                                                          >      >     
>                                                          >      >       
>                                                               be used as
>                                                          >      >     
>                                                          >      >       
>                                                                >       
>                                                                 
>                                                           Management
>                                                         VNI. VXLAN
>                                                          >      >   
>                                                           packet for
>                                                          >      >     
>                                                          >      >       
>                                                               Management
>                                                         VNI MUST NOT
>                                                          >      >     
>                                                          >      >       
>                                                                >       
>                                                                   be
>                                                         sent to a
>                                                         tenant. VNI
>                                                          >      >   
>                                                           number 1 is
>                                                          >      >     
>                                                          >      >       
>                                                             
>                                                           RECOMMENDED as the
>                                                          >      >     
>                                                          >      >       
>                                                                >       
>                                                                 
>                                                           default for
>                                                          >   
>                                                           Management VNI.
>                                                          >      >     
>                                                          >      >       
>                                                                >
>                                                          >      >     
>                                                          >      >       
>                                                                >       
>                                                               With that
>                                                         new text, what
>                                                          >     can be the
>                                                          >      >     
>                                                          >     value of
>                                                          >      >     
>                                                          >      >       
>                                                               the
>                                                         destination
>                                                          >      >     
>                                                          >      >       
>                                                                >       
>                                                               MAC in the
>                                                         inner Ethernet? I
>                                                          >      >   
>                                                           tend to
>                                                          >      >     
>                                                          >     believe
>                                                          >      >     
>                                                          >      >       
>                                                               that it can be
>                                                          >      >     
>                                                          >      >       
>                                                                >       
>                                                               anything
>                                                         and ignored by the
>                                                          >      >   
>                                                           reciever VTEP.
>                                                          >      >     
>                                                          >      >       
>                                                               Also, if the
>                                                          >      >     
>                                                          >      >       
>                                                                >       
>                                                               trapping
>                                                         is based on VNI
>                                                          >      >   
>                                                           number, the
>                                                          >      >     
>                                                          >      >       
>                                                             
>                                                           destination IP
>                                                         address
>                                                          >      >     
>                                                          >      >       
>                                                                >       
>                                                               of the
>                                                         inner IP packet
>                                                          >     can from
>                                                          >      >   
>                                                           the range
>                                                          >      >     
>                                                          >      >       
>                                                               127/8 for
>                                                         IPv4,
>                                                          >      >     
>                                                          >      >       
>                                                                >       
>                                                               and for
>                                                         IPv6 from the range
>                                                          >      >     
>                                                          >      >       
>                                                             
>                                                          >      >     
>                                                          >      >       
>                                                                >       
>                                                               lastly,
>                                                         the TTL to be
>                                                          >      >     
>                                                          >      >     
>                                                          >      >       
>                                                                >
>                                                          >      >     
>                                                          >      >       
>                                                                >       
>                                                          >      >     
>                                                          >      >     
>                                                          >      >       
>                                                          >      >     
>                                                          >      >       
>                                                                >
>                                                          >      >     
>                                                          >      >       
>                                                                >       
>                                                          >      >     
>                                                          >      >       
>                                                                >       
>                                                          >      >     
>                                                          >      >       
>                                                                >
>                                                          >      >     
>                                                          >      >
>                                                          >      >      >
>                                                          >      >

---

From: Anoop Ghanwani <[email-masked]>
Date: Wed, 25 Sep 2019 16:29:12 -0700

Some people may argue that all Tenant Systems connecting to the same
Virtual Network MUST share one VAP, if that's true, then VAP1 and VAP3
should merge into one VAP and my explanation doesn't work. Copying to NVO3
WG to involve more experts, hope for your clarifications and comments.

I would be one of those that would argue that they MUST share on VAP if
they connect to the same Virtual Network.  IMO, the NVO3 arch doc should
have been clearer about this.

Thanks,
Anoop

On Tue, Sep 24, 2019 at 7:40 PM <[email-masked]> wrote:

> Hi Santosh,
>
>
> With regard to the question whether we should allow multiple BFD sessions
> for the same VNI or not, IMHO we should allow it, more explanation as
> follows.
>
> NVE1 and VAP1of NVE2, at the same time another BFD session between VAP3 of
> NVE1 and VAP3 of NVE2, although the two BFD sessions are for the same
> should merge into one VAP and my explanation doesn't work. Copying to NVO3
> WG to involve more experts, hope for your clarifications and comments.
>
>
> Best Regards,
>
> Xiao Min
>

---

From: <[email-masked]>
Date: Thu, 26 Sep 2019 14:20:59 +0800 (CST)

Hi Joel,

Thanks for your comments.

I don't think the VNI could own IP address and MAC address, if the BFD messages are originated and terminated at the VNI, then what addresses would be used by the BFD messages?

As to the VAP, RFC8014 defines it as below:

"On the NVE side, a VAP is a logical network port (virtual or physical) into a specific virtual network."
I interpret this definition as that the VAP could own IP address and MAC address, so I tend to believe the BFD messages are originated and terminated at the VAP.

p.s. I trimmed this mail to meet the size limit, my last mail is too big, which results in a warning from [email-masked].

Best Regards,

Xiao Min

原始邮件

发件人：JoelM.Halpern <[email-masked]>
收件人：肖敏10093570;
抄送人：[email-masked] <[email-masked]>;[email-masked] <[email-masked]>;[email-masked] <[email-masked]>;[email-masked] <[email-masked]>;
日 期 ：2019年09月25日 11:00
主 题 ：Re: [nvo3] BFD over VXLAN: Trapping BFD Control packet at VTEP

Yours,
Joel

>                      |         Data Center Network (IP)        |

_______________________________________________
nvo3 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/nvo3

---

From: <[email-masked]>
Date: Thu, 26 Sep 2019 14:36:15 +0800 (CST)

Hi Anoop,

Considering a scenario where TS1 has an MPLS access (i.e. MPLS-Packet over Tunnel between NVEs) to VNI1, TS3 has an Ethernet access (i.e. MAC-Frame over Tunnel between NVEs) to VNI1, then how can TS1 and TS3 share one VAP?

Best Regards,

Xiao Min

原始邮件

发件人：AnoopGhanwani <[email-masked]>
收件人：肖敏10093570;
抄送人：Greg Mirsky <[email-masked]>;[email-masked] <[email-masked]>;[email-masked] <[email-masked]>;[email-masked] <[email-masked]>;[email-masked] <[email-masked]>;rtg-bfd WG <[email-masked]>;Joel M. Halpern <[email-masked]>;[email-masked] <[email-masked]>;[email-masked] <[email-masked]>;
日 期 ：2019年09月26日 08:36
主 题 ：Re: [nvo3] BFD over VXLAN: Trapping BFD Control packet at VTEP

I would be one of those that would argue that they MUST share on VAP if they connect to the same Virtual Network.  IMO, the NVO3 arch doc should have been clearer about this.

Thanks,
Anoop

Hi Santosh,

With regard to the question whether we should allow multiple BFD sessions for the same VNI or not, IMHO we should allow it, more explanation as follows...

Best Regards,

Xiao Min

---

From: Anoop Ghanwani <[email-masked]>
Date: Thu, 26 Sep 2019 08:16:30 -0700

Hi Xiao Min,

I think we would need more detail around the use case below.  What does the
MPLS packet over Tunnel look like?

Thanks,
Anoop

On Wed, Sep 25, 2019 at 11:37 PM <[email-masked]> wrote:

> Considering a scenario where TS1 has an MPLS access (i.e. MPLS-Packet over
> Tunnel between NVEs) to VNI1, TS3 has an Ethernet access (i.e. MAC-Frame
> over Tunnel between NVEs) to VNI1, then how can TS1 and TS3 share one VAP?
>
>
> Best Regards,
>
> Xiao Min
> 原始邮件
> *发件人：*AnoopGhanwani <[email-masked]>
> *收件人：*肖敏10093570;
> *抄送人：*Greg Mirsky <[email-masked]>;[email-masked] <
> [email-masked]>;[email-masked] <
> [email-masked]>;[email-masked] <[email-masked]>;
> [email-masked] <[email-masked]>;rtg-bfd WG <
> [email-masked]>;Joel M. Halpern <[email-masked]>;
> [email-masked] <[email-masked]>;[email-masked] <
> [email-masked]>;
> *日 期 ：*2019年09月26日 08:36
> *主 题 ：**Re: [nvo3] BFD over VXLAN: Trapping BFD Control packet at VTEP*
>> sessions for the same VNI or not, IMHO we should allow it, more explanation
>> as follows...
>>

---

From: <[email-masked]>
Date: Fri, 27 Sep 2019 10:49:03 +0800 (CST)

Hi Anoop,

Due to the fact that a variety of Tunnels could be used under the NVO3 architecture, as an example, below figure illustrates the format of MPLS packet over Geneve Tunnel.

 0 1 2 3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 | |
 ~ Outer Ethernet Header ~
 | |
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 | |
 ~ Outer IPvX Header ~
 | |
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 | |
 ~ Outer UDP Header ~
 | |
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 | |
 ~ Geneve Header ~
 | |
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+<-+
 | | |
 ~ MPLS Label Stack ~ M
 | | P
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ L
 | | S
 | |
 ~ Payload ~ P
 | | K
 | | T
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+<-+
 | FCS |
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Note that in NVO3 working group Greg and I have submitted an individual draft draft-xiao-nvo3-bfd-geneve, which is used to address BFD over Geneve.

The intention is to make the two drafts draft-ietf-bfd-vxlan and draft-xiao-nvo3-bfd-geneve aligned, that is to say, we try to define the identical mechanism for the common part of BFD over VxLAN Tunnel and BFD over Geneve Tunnel. For the common part, draft-xiao-nvo3-bfd-geneve would reference to draft-ietf-bfd-vxlan, and for the other part specific to Geneve, we'll define the specific mechanism in draft-xiao-nvo3-bfd-geneve.

Hope that clarifies.

Best Regards,

Xiao Min

原始邮件

日 期 ：2019年09月26日 23:16

Hi Xiao Min,
I think we would need more detail around the use case below.  What does the MPLS packet over Tunnel look like?

Thanks,
Anoop

Hi Anoop,

Best Regards,

Xiao Min

原始邮件

Thanks,
Anoop

Hi Santosh,

Best Regards,

Xiao Min

---

From: Anoop Ghanwani <[email-masked]>
Date: Fri, 27 Sep 2019 14:35:33 -0700

Hi Xiao Min,

Thanks for the details about the encap but the use case is not clear.  It
might help if you explain why its necessary to map a physical Ethernet port
and/or 802.1Q VLAN to the same VNI as an MPLS packet without an L2 header.

Thanks,
Anoop

On Thu, Sep 26, 2019 at 7:50 PM <[email-masked]> wrote:

> Hi Anoop,
>
>
> Due to the fact that a variety of Tunnels could be used under the NVO3 architecture,
> as an example, below figure illustrates the format of MPLS packet over
> Geneve Tunnel.
>
>     0                   1                   2                   3
> Note that in NVO3 working group Greg and I have submitted an individual
> draft draft-xiao-nvo3-bfd-geneve, which is used to address BFD over Geneve.
>
> The intention is to make the two drafts draft-ietf-bfd-vxlan and
> draft-xiao-nvo3-bfd-geneve aligned, that is to say, we try to define the
> identical mechanism for the common part of BFD over VxLAN Tunnel and BFD
> over Geneve Tunnel. For the common part, draft-xiao-nvo3-bfd-geneve would
> reference to draft-ietf-bfd-vxlan, and for the other part specific to
> Geneve, we'll define the specific mechanism in draft-xiao-nvo3-bfd-geneve.
>
>
> Hope that clarifies.
>
>
> Best Regards,
>
> Xiao Min
> 原始邮件
> *日 期 ：*2019年09月26日 23:16
>> Considering a scenario where TS1 has an MPLS access (i.e. MPLS-Packet
>> over Tunnel between NVEs) to VNI1, TS3 has an Ethernet access (i.e.
>> MAC-Frame over Tunnel between NVEs) to VNI1, then how can TS1 and TS3 share
>> one VAP?
>>
>>
>> Best Regards,
>>
>> Xiao Min
>> 原始邮件