Subject: Re: [nvo3] Security in VXLAN-GPE (was Re: Consensus call on encap proposals)

From: Fabio Maino <[email-masked]>
Date: Mon, 1 Aug 2016 14:59:45 -0700

On 7/29/16 6:38 PM, Jesse Gross wrote:
>> On Jul 29, 2016, at 4:39 PM, Fabio Maino <[email-masked]> wrote:
>>
>> On 7/29/16 12:44 PM, Jesse Gross wrote:
>>>> On Jul 29, 2016, at 12:17 PM, Fabio Maino <[email-masked]> wrote:
>>>>
>>>> On 7/29/16 11:45 AM, Tom Herbert wrote:
>>>>> On Jul 29, 2016 11:12 AM, "Fabio Maino" <[email-masked]> wrote:
>>>>>> On 7/27/16 1:43 PM, Tom Herbert wrote:
>>>>>>> On Wed, Jul 27, 2016 at 1:15 PM, Fabio Maino <[email-masked]> wrote:
>>>>>>>> On 7/27/16 12:27 PM, Tom Herbert wrote:
>>>>>>>>> On Wed, Jul 27, 2016 at 11:00 AM, Fabio Maino <[email-masked]> wrote:
>>>>>>>>>> On 7/27/16 10:53 AM, Tom Herbert wrote:
>>>>>>>>>>> On Wed, Jul 27, 2016 at 10:44 AM, Dino Farinacci <[email-masked]>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>> On Jul 26, 2016, at 3:11 PM, Fabio Maino (fmaino) <[email-masked]>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>> On 7/26/16 12:08 PM, Dino Farinacci wrote:
>>>>>>>>>>>>>>>> Having VXLAN as an Independent Stream RFC gives a document to be
>>>>>>>>>>>>>>>> used
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>> innovate from.   I believe that was the intent of VXLAN-GPE - to
>>>>>>>>>>>>>>>> provide the ability
>>>>>>>>>>>>>>>> for needed extensions.
>>>>>>>>>>>>>>> The only new feature the VXLAN-GPE brings to the table is a way to
>>>>>>>>>>>>>>> identify that NSH is being encapsulated.
>>>>>>>>>>>>>> or any other shim header, NSH just happens to be one.
>>>>>>>>>>>>> Precisely.  GPE addresses the limitation associated with VXLAN, namely
>>>>>>>>>>>>> the lack of multi-protocol support.  As we've seen on the list, other
>>>>>>>>>>>>> protocols have come up: MPLS for example.
>>>>>>>>>>>> This is technically not true. As long as you are willing to spend 14
>>>>>>>>>>>> bytes on a MAC header, then anything with an ethertype can be
>>>>>>>>>>>> encapsulated
>>>>>>>>>>>> with VXLAN.
>>>>>>>>>>>>
>>>>>>>>>>>> And I have working code that encapsulates both IPv4 and IPv6 in VXLAN
>>>>>>>>>>>> with ethertypes 0x0800 and 0x86dd, respectively.
>>>>>>>>>>>>
>>>>>>>>>>>>> There are some other technical benefits as well: version and OAM.
>>>>>>>>>>> As do Geneve and GUE. But this thread is not about touting the
>>>>>>>>>>> features of these protocols, it is about the technical objections to
>>>>>>>>>>> them. My major objections (from the list I posted) to VXLAN-GPE is
>>>>>>>>>>> that it has no extensibility and offers no security of the VNI. These
>>>>>>>>>>> are showstoppers in my deployment.
>>>>>>>>>>
>>>>>>>>>> Tom,
>>>>>>>>>>
>>>>>>>>>> extensibility is achieved via shim header:
>>>>>>>>>>
>>>>>>>>>> - if you want to carry end to end metadata you can define the appropriate
>>>>>>>>>> shim header
>>>>>>>>>>
>>>>>>>>>> - If you want to protect the integrity of the VNI, you can include an
>>>>>>>>>> HMAC
>>>>>>>>>> of the VNI in the shim header
>>>>>>>>>>
>>>>>>>>> Please point me to the specification where this shim header with the
>>>>>>>>> HMAC is described and add a reference to it in the security
>>>>>>>>> considerations section of the VXLAN-GPE draft.
>>>>>>>> Hi Tom,
>>>>>>>> there are a lot of use cases that can be addressed with shim headers, and
>>>>>>>> each solution may look fairly different depending on the requirements.
>>>>>>>>
>>>>>>> The security option including the specific header format is described
>>>>>>> in draft-herbert-gue-extensions. You can use the same data format.
>>>>>>>
>>>>>>>> Can you point me to an nvo3 specification of what are the requirements that
>>>>>>>> need to be addressed to provide VNI integrity protection? If the
>>>>>>>> requirements are the same you used for GUE, the shim header could use an
>>>>>>>> encoding similar to the GUE security option.
>>>>>>>>
>>>>>>> Section 6.2 of draft-ietf-nvo3-security-requirements describes the
>>>>>>> dataplane requirements. The most important requirement of nvo3 is to
>>>>>>> maintain strict isolation between tenant virtual networks. The problem
>>>>>>> is exacerbated in UDP encapsulation since any non privileged
>>>>>>> application can send a UDP packet to any port thereby making it easier
>>>>>>> to inject packets into a virtual network than in a non-UDP based encap
>>>>>>> protocol like nvgre. In our large scale network we need strong
>>>>>>> mechanisms to guarantee this isolation; Ethernet CRC and network
>>>>>>> mechanisms like firewalls or address spoofing are not sufficient for
>>>>>>> multi-tenant nv security.
>>>>>> Tom,
>>>>>> I went back and re-read the draft. Requirement 10, that is the one that seems relevant here, is not a MUST. From that standpoint VXLAN-GPE does comply with that draft.
>>>>>>
>>>>>> I don't want to argue that there are use cases where authenticaton of the VNI might be useful, but certainly there are others where it isn't. Otherwise the WG would have settled for a MUST there.
>>>>>>
>>>>>> In the design of VXLAN-GPE we have choosen to not burden the cost of ALL the implementations with a mandatory authentication field. However, thanks to VXLAN-GPE flexibility and extensibility, this function can be added via shim header.
>>>>>>
>>>>>>
>>>>>>>> Lacking the requirement document, I'm sure we can add a paragraph in the
>>>>>>>> security section that describes how it can be done.
>>>>>>>>
>>>>>>> Please be specific in that. If you just say "use NSH" that is not
>>>>>>> helpful at all to your potential implementors or users. If we were to
>>>>>>> implement security we need to know the exact bits that are set in the
>>>>>>> packet.
>>>>>>>
>>>>>>> Tom
>>>>>>
>>>>>> That should not go in the VXLAN-GPE draft. All that draft needs to specify is that there might be a following shim header. The VXLAN-GPE draft does that today.
>>>>>>
>>>>>> if I read your comment right, you seem to agree that it can be done. Is this your MAJOR objection to VXLAN-GPE? If we were to specify such a draft, would that change your opinion in supporting VXLAN-GPE for WG adoption?
>>>>>>
>>>>> My major objections are lack of security and extensibility. The counter argument seems to be use a shim header for security or extensions, but there has never be any specifics offered (I have asked several times now how to implement a 64 bit security cookie in VXLAN-GPE). Also the security considerations section in the draft provides no useful guidance.
>>>>>
>>>>> In my data center if we are going to commit to using a protocol for the long term (like if we buy into HW support), I need to believe it is adaptable to face changing security risks and new technologies. I know how to extend GUE and in fact we have useful extensions defined. I believe I would know how to extend Geneve, although I still wish there were some standard TLVs spec'ed. However, I don't know how extend VXLAN-GPE. I can't afford to find out two years from now that I need switch protocols because we can't adapt the running protocol. This is particularly true in data center security, the demands for strong security mechanisms will only be increasing-- I need a protocol that can deal with that...
>>>>>
>>>>> This is all just my opinion of course :-)
>>>>>
>>>> I think that the point at hand here is that the AD is pushing the WG to have a discussion on which encapsulations should be selected for standard track. What should guide that decision are the requirements emerging from the set of documents that are adopted by the WG.
>>>>
>>>> As I said in the previous message, IMO VXLAN-GPE does address the security requirements as expressed in draft-ietf-nvo3-security-requirements. I note that you have not objected to that point.
>>>>
>>>> Also, I pointed to NSH as ONE example of how VXLAN-GPE can be extended using a shim header. NSH implements SFC and metadata transport. Looking at what NSH + VXLAN-GPE achieves, it makes fairly hard to maintain that VXLAN-GPE is not extensible. Extensibility is achieved via layering. It might not be the same way extensibility is achieved by the other contenders, but it's a fairly established way to design protocols.
>>>>
>>>> I think this should answer the objections to security and extensibility for the purpose of the selection of an nvo3 encapsulation.
>>> The issue with shim headers is that introducing a new one breaks the ability of existing implementations to parse the packet. There are a lot of use cases where just being able to get to the packet data is hugely useful - a NIC steering incoming frames to a VM shouldn’t need to be modified for every use case. Even simple examples like tcpdump/Wireshark benefit from this - it’s really useful to be able to generally get a look at a packet even if it doesn’t understand everything. With both Geneve and GUE this is possible but not with the design that you are describing with VXLAN-GPE.
>> That approach forces your customers to pay upfront for the cost of buffering metadata that their devices don't know how to handle, and/or they may never use. To implement a new feature, you'll need buffers AND logic.
>>
>> Even if you allocate buffer in advance (before the feature is specified), you'll have to spin a new version of the HW to add the logic that addresses that new feature.
>>
>> In this sense VXLAN-GPE allows an implementor to be sensible in allocating resources, by using only the buffer space that is actually needed by that specific implementation.
>>
>> I suspect that the result of the pressure of cost on implementations will be such that, some HW implementations of Geneve, may implement only the basic functions,  restricting to very few, if any, support for variable length options. Packets with unknown options will be eventually punted to the slow path, and vendors will claim support for the protocol.
>>
>> This is not very different from how an unknown shim header will be handled. Flexibility and HW implementations might be a bit of a unicorn…
> I think that you are making some specific assumptions about the types of implementations here. In particular, you are assuming that the only device class is a fixed-function switching ASIC. The comments don’t really apply to software, NICs, or even the more programmable switching ASICs that are starting to come out.
>
> In that specific situation, I agree that it doesn’t make sense to include elements of an implementation that can’t be usefully deployed. However, in cases where the device would need to be respun to include new logic, I doubt that anyone would prematurely include vestigal pieces that would drive up the cost. They would simply implement the support as necessary and the cost would reflect what the device is capable of doing. This is similar to VXLAN-GPE in that regard.
>
> However, I don’t believe that networks consist exclusively of fixed switching ASICs. As I mentioned before, there are many other components that can take advantage of extensibility. I don’t see that it is necessary to deny them those benefits even if some devices may need to be incrementally respun.

This diverse base of implementation platforms is, IMO, what makes hard 
to select a single encap for the nvo3 use case.  A feature of the 
protocol that might be a benefit for one platform, introduces 
significant costs/disadvantages when deployed in another one.

Fabio

>
>>> What you are describing is essentially the same as IPv6 extension headers vs. IPv4 options. Anecdotally, I’ve seen a lot more bugs in implementations of IPv6 extension headers even in very common cases. The reason is that with IPv6, it’s necessary to traverse different shim headers, which themselves have different locations of next protocol field is, etc. At least with IPv4, getting to the data is very simple and consistent.
>> This is actually a good example for the counter argument: many vendors couldn't justify the cost of supporting IPv4 option in the fast path, and packet with IP options, in many implementations, were punted to the slow path. This quickly drove developers to abandon use of IP options in their applications, as it was almost a guarantee to send their flows in the slow path. Some time too much flexibility is bad (especially when cost is an issue).
>>
>> For IPv6 extension headers, use cases will determine (are determining) which ones are worth the extra cost in the fast path and vendors will act accordingly. Fittest options will survive…
> Yes, I certainly agree that vendors will act according to the needs that they see in the market. I think it’s been said a number of times in the working group that extensibility is a “use it or lose it” type of thing. Since IP options didn’t have immediate important uses when it was first being implemented, there was little sense in handling them. In contrast, TCP options are alive and well these days because there are good uses for them. I know that TCP is generally implemented in software instead of hardware but I can assure you the desire to take shortcuts is plenty strong in the software industry.
>
> In the specific case of Geneve, I gave some information about some of the implementations I’ve personally seen and their capabilities a while back (https://mailarchive.ietf.org/arch/msg/nvo3/HTSXROpKwman1Bd4fAlmmfZF_P8). Since options are the core reason for Geneve existing (it’s the main thing that separates it from VXLAN), there’s been much more awareness that it is important and useful to implement them.
>
>>> To make this very concrete: I know that there a number of NIC vendors that have implemented support for VXLAN-GPE offloading already. If a shim header was introduced to add VNI security as you are describing, these offloads would suddenly become useless. That does not seem like a very good definition of extensibility to me.
>>
>> Similarly to those vendors that might have implemented Geneve in the fast path, and want to add support for a new optional extension. With the difference that their customers will have paid already for the cost of extra buffering even in the first release that had no support for that new option.
> In the specific example that I gave (NICs), that is not really true. As new Geneve options are introduced the existing cards will continue to work (GUE also has this property). This is because these offloads are primarily operating on the payload while the option logic is handled in the host CPU. For server-based tunnel termination this ability to continue to offload as the protocol evolves is a significant benefit.

---

From: Tom Herbert <[email-masked]>
Date: Mon, 1 Aug 2016 15:35:22 -0700

On Mon, Aug 1, 2016 at 2:59 PM, Fabio Maino <[email-masked]> wrote:
>>>>>>> I went back and re-read the draft. Requirement 10, that is the one
>>>>>>> that seems relevant here, is not a MUST. From that standpoint VXLAN-GPE does
>>>>>>> comply with that draft.
>>>>>>>
>>>>>>> I don't want to argue that there are use cases where authenticaton of
>>>>>>> the VNI might be useful, but certainly there are others where it isn't.
>>>>>>> Otherwise the WG would have settled for a MUST there.
>>>>>>>
>>>>>>> In the design of VXLAN-GPE we have choosen to not burden the cost of
>>>>>>> ALL the implementations with a mandatory authentication field. However,
>>>>>>> thanks to VXLAN-GPE flexibility and extensibility, this function can be
>>>>>>> added via shim header.
>>>>>>>
>>>>>>>
>>>>>>> That should not go in the VXLAN-GPE draft. All that draft needs to
>>>>>>> specify is that there might be a following shim header. The VXLAN-GPE draft
>>>>>>> does that today.
>>>>>>>
>>>>>>> if I read your comment right, you seem to agree that it can be done.
>>>>>>> Is this your MAJOR objection to VXLAN-GPE? If we were to specify such a
>>>>>>> draft, would that change your opinion in supporting VXLAN-GPE for WG
>>>>>>> adoption?
>>>>>>>
>>>>>> My major objections are lack of security and extensibility. The
>>>>>> counter argument seems to be use a shim header for security or extensions,
>>>>>> but there has never be any specifics offered (I have asked several times now
>>>>>> how to implement a 64 bit security cookie in VXLAN-GPE). Also the security
>>>>>> considerations section in the draft provides no useful guidance.
>>>>>>
>>>>>> In my data center if we are going to commit to using a protocol for
>>>>>> the long term (like if we buy into HW support), I need to believe it is
>>>>>> adaptable to face changing security risks and new technologies. I know how
>>>>>> to extend GUE and in fact we have useful extensions defined. I believe I
>>>>>> would know how to extend Geneve, although I still wish there were some
>>>>>> standard TLVs spec'ed. However, I don't know how extend VXLAN-GPE. I can't
>>>>>> afford to find out two years from now that I need switch protocols because
>>>>>> we can't adapt the running protocol. This is particularly true in data
>>>>>> center security, the demands for strong security mechanisms will only be
>>>>>> increasing-- I need a protocol that can deal with that...
>>>>>>
>>>>> I think that the point at hand here is that the AD is pushing the WG to
>>>>> have a discussion on which encapsulations should be selected for standard
>>>>> track. What should guide that decision are the requirements emerging from
>>>>> the set of documents that are adopted by the WG.
>>>>>
>>>>> As I said in the previous message, IMO VXLAN-GPE does address the
>>>>> security requirements as expressed in draft-ietf-nvo3-security-requirements.
>>>>> I note that you have not objected to that point.
>>>>>
>>>>> Also, I pointed to NSH as ONE example of how VXLAN-GPE can be extended
>>>>> using a shim header. NSH implements SFC and metadata transport. Looking at
>>>>> what NSH + VXLAN-GPE achieves, it makes fairly hard to maintain that
>>>>> VXLAN-GPE is not extensible. Extensibility is achieved via layering. It
>>>>> might not be the same way extensibility is achieved by the other contenders,
>>>>> but it's a fairly established way to design protocols.
>>>>>
>>>>> I think this should answer the objections to security and extensibility
>>>>> for the purpose of the selection of an nvo3 encapsulation.
>>>>
>>>> The issue with shim headers is that introducing a new one breaks the
>>>> ability of existing implementations to parse the packet. There are a lot of
>>>> use cases where just being able to get to the packet data is hugely useful -
>>>> a NIC steering incoming frames to a VM shouldn’t need to be modified for
>>>> every use case. Even simple examples like tcpdump/Wireshark benefit from
>>>> this - it’s really useful to be able to generally get a look at a packet
>>>> even if it doesn’t understand everything. With both Geneve and GUE this is
>>>> possible but not with the design that you are describing with VXLAN-GPE.
>>>
>>> That approach forces your customers to pay upfront for the cost of
>>> buffering metadata that their devices don't know how to handle, and/or they
>>> may never use. To implement a new feature, you'll need buffers AND logic.
>>>
>>> Even if you allocate buffer in advance (before the feature is specified),
>>> you'll have to spin a new version of the HW to add the logic that addresses
>>> that new feature.
>>>
>>> In this sense VXLAN-GPE allows an implementor to be sensible in
>>> allocating resources, by using only the buffer space that is actually needed
>>> by that specific implementation.
>>>
>>> I suspect that the result of the pressure of cost on implementations will
>>> be such that, some HW implementations of Geneve, may implement only the
>>> basic functions,  restricting to very few, if any, support for variable
>>> length options. Packets with unknown options will be eventually punted to
>>> the slow path, and vendors will claim support for the protocol.
>>>
>>> This is not very different from how an unknown shim header will be
>>> handled. Flexibility and HW implementations might be a bit of a unicorn…
>>
>> I think that you are making some specific assumptions about the types of
>> implementations here. In particular, you are assuming that the only device
>> class is a fixed-function switching ASIC. The comments don’t really apply to
>> software, NICs, or even the more programmable switching ASICs that are
>> starting to come out.
>>
>> In that specific situation, I agree that it doesn’t make sense to include
>> elements of an implementation that can’t be usefully deployed. However, in
>> cases where the device would need to be respun to include new logic, I doubt
>> that anyone would prematurely include vestigal pieces that would drive up
>> the cost. They would simply implement the support as necessary and the cost
>> would reflect what the device is capable of doing. This is similar to
>> VXLAN-GPE in that regard.
>>
>> However, I don’t believe that networks consist exclusively of fixed
>> switching ASICs. As I mentioned before, there are many other components that
>> can take advantage of extensibility. I don’t see that it is necessary to
>> deny them those benefits even if some devices may need to be incrementally
>> respun.
>
>
> This diverse base of implementation platforms is, IMO, what makes hard to
> select a single encap for the nvo3 use case.  A feature of the protocol that
> might be a benefit for one platform, introduces significant
> costs/disadvantages when deployed in another one.
>
Can you give a specific example of a feature that demonstrates this?
With regards to extensibility I am not seeing why using NSH has lower
costs or fewer disadvantages to deploy compared to flag-fields in GUE
or even TLVs in Geneve...

Tom

> Fabio
>
>
>
>>
>>>> What you are describing is essentially the same as IPv6 extension
>>>> headers vs. IPv4 options. Anecdotally, I’ve seen a lot more bugs in
>>>> implementations of IPv6 extension headers even in very common cases. The
>>>> reason is that with IPv6, it’s necessary to traverse different shim headers,
>>>> which themselves have different locations of next protocol field is, etc. At
>>>> least with IPv4, getting to the data is very simple and consistent.
>>>
>>> This is actually a good example for the counter argument: many vendors
>>> couldn't justify the cost of supporting IPv4 option in the fast path, and
>>> packet with IP options, in many implementations, were punted to the slow
>>> path. This quickly drove developers to abandon use of IP options in their
>>> applications, as it was almost a guarantee to send their flows in the slow
>>> path. Some time too much flexibility is bad (especially when cost is an
>>> issue).
>>>
>>> For IPv6 extension headers, use cases will determine (are determining)
>>> which ones are worth the extra cost in the fast path and vendors will act
>>> accordingly. Fittest options will survive…
>>
>> Yes, I certainly agree that vendors will act according to the needs that
>> they see in the market. I think it’s been said a number of times in the
>> working group that extensibility is a “use it or lose it” type of thing.
>> Since IP options didn’t have immediate important uses when it was first
>> being implemented, there was little sense in handling them. In contrast, TCP
>> options are alive and well these days because there are good uses for them.
>> I know that TCP is generally implemented in software instead of hardware but
>> I can assure you the desire to take shortcuts is plenty strong in the
>> software industry.
>>
>> In the specific case of Geneve, I gave some information about some of the
>> implementations I’ve personally seen and their capabilities a while back
>> (https://mailarchive.ietf.org/arch/msg/nvo3/HTSXROpKwman1Bd4fAlmmfZF_P8).
>> Since options are the core reason for Geneve existing (it’s the main thing
>> that separates it from VXLAN), there’s been much more awareness that it is
>> important and useful to implement them.
>>
>>>> To make this very concrete: I know that there a number of NIC vendors
>>>> that have implemented support for VXLAN-GPE offloading already. If a shim
>>>> header was introduced to add VNI security as you are describing, these
>>>> offloads would suddenly become useless. That does not seem like a very good
>>>> definition of extensibility to me.
>>>
>>>
>>> Similarly to those vendors that might have implemented Geneve in the fast
>>> path, and want to add support for a new optional extension. With the
>>> difference that their customers will have paid already for the cost of extra
>>> buffering even in the first release that had no support for that new option.
>>
>> In the specific example that I gave (NICs), that is not really true. As
>> new Geneve options are introduced the existing cards will continue to work
>> (GUE also has this property). This is because these offloads are primarily
>> operating on the payload while the option logic is handled in the host CPU.
>> For server-based tunnel termination this ability to continue to offload as
>> the protocol evolves is a significant benefit.
>
>
>

---

From: Alia Atlas <[email-masked]>
Date: Mon, 1 Aug 2016 19:21:22 -0400

On Mon, Aug 1, 2016 at 6:35 PM, Tom Herbert <[email-masked]> wrote:

> >>>>>>>>>> On Wed, Jul 27, 2016 at 11:00 AM, Fabio Maino <[email-masked]
> >
> >>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Having VXLAN as an Independent Stream RFC gives a
> document
> >>>>>>>>>>>>>>>>> to be
> >>>>>>>>>>>>>>>>> used
> >>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>> innovate from.   I believe that was the intent of
> VXLAN-GPE
> >>>>>>>>>>>>>>>>> - to
> >>>>>>>>>>>>>>>>> provide the ability
> >>>>>>>>>>>>>>>>> for needed extensions.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> The only new feature the VXLAN-GPE brings to the table is
> a
> >>>>>>>>>>>>>>>> way to
> >>>>>>>>>>>>> And I have working code that encapsulates both IPv4 and IPv6
> in
> >>>>>>>>>>>>> VXLAN
> >>>>>>>>>>>> them. My major objections (from the list I posted) to
> VXLAN-GPE
> >>>>>>>>>>>> is
> >>>>>>>>>>>> that it has no extensibility and offers no security of the
> VNI.
> >>>>>>>>>>>> These
[Alia] Do recall that the question is around significant technical concerns
that
must be addressed.   It isn't whether one approach is "better" or "easier"
for a
particular deployment scenario as compared to another.

Regards,
Alia

> _______________________________________________
> nvo3 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nvo3
>

---

From: Fabio Maino <[email-masked]>
Date: Wed, 3 Aug 2016 14:26:33 -0700

On 8/1/16 4:21 PM, Alia Atlas wrote:
>
>
> On Mon, Aug 1, 2016 at 6:35 PM, Tom Herbert <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>     On Mon, Aug 1, 2016 at 2:59 PM, Fabio Maino <[email-masked]
>     <mailto:[email-masked]>> wrote:
>     >>> On Jul 29, 2016, at 4:39 PM, Fabio Maino <[email-masked]
>     <mailto:[email-masked]>> wrote:
>     >>>
>     >>>>> On Jul 29, 2016, at 12:17 PM, Fabio Maino <[email-masked]
>     <mailto:[email-masked]>> wrote:
>     >>>>>
>     >>>>>> On Jul 29, 2016 11:12 AM, "Fabio Maino" <[email-masked]
>     <mailto:[email-masked]>> wrote:
>     >>>>>>>
>     >>>>>>>> On Wed, Jul 27, 2016 at 1:15 PM, Fabio Maino
>     <[email-masked] <mailto:[email-masked]>>
>     >>>>>>>> wrote:
>     >>>>>>>>>
>     >>>>>>>>>> On Wed, Jul 27, 2016 at 11:00 AM, Fabio Maino
>     <[email-masked] <mailto:[email-masked]>>
>     >>>>>>>>>> wrote:
>     >>>>>>>>>>>
>     >>>>>>>>>>>> On Wed, Jul 27, 2016 at 10:44 AM, Dino Farinacci
>     >>>>>>>>>>>> <[email-masked] <mailto:[email-masked]>>
>     >>>>>>>>>>>> wrote:
>     >>>>>>>>>>>>>>
>     >>>>>>>>>>>>>> On Jul 26, 2016, at 3:11 PM, Fabio Maino (fmaino)
>     >>>>>>>>>>>>>> <[email-masked] <mailto:[email-masked]>>
>     >>>>>>>>>>>>>> wrote:
>     >>>>>>>>>>>>>>>
>     >>>>>>>>>>>>>>>>> Having VXLAN as an Independent Stream RFC gives
>     a document
>     >>>>>>>>>>>>>>>>> to be
>     >>>>>>>>>>>>>>>>> used
>     >>>>>>>>>>>>>>>>> to
>     >>>>>>>>>>>>>>>>> innovate from.   I believe that was the intent
>     of VXLAN-GPE
>     >>>>>>>>>>>>>>>>> - to
>     >>>>>>>>>>>>>>>>> provide the ability
>     >>>>>>>>>>>>>>>>> for needed extensions.
>     >>>>>>>>>>>>>>>>
>     >>>>>>>>>>>>>>>> The only new feature the VXLAN-GPE brings to the
>     table is a
>     >>>>>>>>>>>>>>>> way to
>     >>>>>>>>>>>>>> Precisely.  GPE addresses the limitation associated
>     with
>     >>>>>>>>>>>>>> VXLAN, namely
>     >>>>>>>>>>>>>> the lack of multi-protocol support.  As we've seen
>     on the
>     >>>>>>>>>>>>>> list, other
>     >>>>>>>>>>>>> This is technically not true. As long as you are
>     willing to
>     >>>>>>>>>>>>> spend 14
>     >>>>>>>>>>>>> And I have working code that encapsulates both IPv4
>     and IPv6 in
>     >>>>>>>>>>>>> VXLAN
>     >>>>>>>>>>>>>> There are some other technical benefits as well:
>     version and
>     >>>>>>>>>>>>>> OAM.
>     >>>>>>>>>>>>
>     >>>>>>>>>>>> that it has no extensibility and offers no security
>     of the VNI.
>     >>>>>>>>>>>> These
>     >>>>>>>>>>> - if you want to carry end to end metadata you can
>     define the
>     >>>>>>>>>>> appropriate
>     >>>>>>>>>>> shim header
>     >>>>>>>>>>>
>     >>>>>>>>>> Please point me to the specification where this shim
>     header with
>     >>>>>>>>>> the
>     >>>>>>>>> requirements are the same you used for GUE, the shim
>     header could
>     >>>>>>>>> use an
>     >>>>>>>> maintain strict isolation between tenant virtual
>     networks. The
>     >>>>>>>> problem
>     >>>>>>>> application can send a UDP packet to any port thereby
>     making it
>     >>>>>>>> easier
>     >>>>>>>> to inject packets into a virtual network than in a
>     non-UDP based
>     >>>>>>>> encap
>     >>>>>>>>> Lacking the requirement document, I'm sure we can add a
>     paragraph
>     >>>>>>>>> in the
>     >>>>>>>> helpful at all to your potential implementors or users.
>     If we were
>     >>>>>>>> to
>     >>>>>>>> implement security we need to know the exact bits that
>     are set in
>     >>>>>>>> the
>     >>>>>>>> packet.
>     >>>>>>>>
>     >>>>>>>> Tom
>     >>>>>>>
>     >>>>>>>
> [Alia] Do recall that the question is around significant technical 
> concerns that
> must be addressed.   It isn't whether one approach is "better" or 
> "easier" for a

The major concern comes from selecting a single MUST implement 
encapsulation that has the option to include hundreds of bytes of 
unstructured, variable length metadata. Such a feature is fairly 
unfriendly to implement in fixed switching ASICs.

Considerations on implementability of an approach is, in my opinion, a 
very significant technical concern in selecting a standard.

Fabio

>     [email-masked] <mailto:[email-masked]>

---

From: Tom Herbert <[email-masked]>
Date: Wed, 3 Aug 2016 15:38:03 -0700

On Wed, Aug 3, 2016 at 2:26 PM, Fabio Maino <[email-masked]> wrote:
>> >>>>>>>>>> <[email-masked]>
>> >>>>>>>>>>>> <[email-masked]>
>> >>>>>>>>>>>>>> <[email-masked]>
>> >>>>>>>>>>>>>> There are some other technical benefits as well: version
>> >>>>>>>>>>>>>> and
>> >>>>>>>>>>>>>> OAM.
>> >>>>>>>>>>>>
>> >>>>>>>>>> Please point me to the specification where this shim header
>> >>>>>>>>>> with
>> >>>>>>>>>> the
>> >>>>>>>>> requirements are the same you used for GUE, the shim header
>> >>>>>>>>> could
>> >>>>>>>>> use an
>> >>>>>>>> helpful at all to your potential implementors or users. If we
>> >>>>>>>> were
>> >>>>>>>> to
> The major concern comes from selecting a single MUST implement encapsulation
> that has the option to include hundreds of bytes of unstructured, variable
> length metadata. Such a feature is fairly unfriendly to implement in fixed
> switching ASICs.
>
This is where I think there is confusion on the "HW implementability"
arguments that are being made. VXLAN-GPE is a fixed length header and
that is perfectly amenable to switching ASICs-- but so is GUE and
Geneve without any options present (they all resolve to a simple fixed
length 8 byte headers that can no doubt be implemented in switching
ASICs with equal ease). But, if you take into account processing
options for extensibility which would be TLVs in Geneve, flag-fields
in GUE, and NSH with VXLAN-GPE as you suggested-- then GUE is the most
friendly to HW (parsing is amenable to implement by TCAM for
instance), and Geneve is likely easier to process since TLVs in one
protocol header is more straightforward for processing a header chain
(e.g. IP options vs. ext. hdrs. experience).

Tom

---

From: Fabio Maino <[email-masked]>
Date: Wed, 3 Aug 2016 15:54:58 -0700

> (e.g. IP options vs. ext. hdrs. experience).

The amount of buffering required, and the number and ordering of options 
makes a difference as well. TCAM adds to the cost and is a resource 
contended by other functions in the ASIC. Given that the HW needs to be 
re-spun anyway to process new features the flexibility of the protocol 
itself doesn't make much of a difference.

Fabio

---

From: Tom Herbert <[email-masked]>
Date: Wed, 3 Aug 2016 16:34:23 -0700

On Wed, Aug 3, 2016 at 3:54 PM, Fabio Maino <[email-masked]> wrote:
>>>>>>>>>>>> dataplane requirements. The most important requirement of nvo3
>>>>>>>>>>>> is
>>>>>>>>>>>> to
>>>>>>>>>>> I don't want to argue that there are use cases where
>>>>>>>>>>> authenticaton
>>>>>>>>>>> of
>>>>>>>>>>>> implement security we need to know the exact bits that are set
>>>>>>>>>>>> in
>>>>>>>>>>>> the
>>>>>>>>>>>> packet.
>>>>>>>>>>>>
>>>>>>>>>>>> Tom
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>> I think that the point at hand here is that the AD is pushing the
>>>>>>>>> WG
>>>>>>>>> to
>>>>>>>>> track. What should guide that decision are the requirements
>>>>>>>>> emerging
>>>>>>>>> from
>>>>>>>>> VXLAN-GPE is not extensible. Extensibility is achieved via
>>>>>>>>> layering.
>>>>>>>>> It
>>>>>>> buffering metadata that their devices don't know how to handle,
>>>>>>> and/or
>>>>>>> they
>>>>>>> length options. Packets with unknown options will be eventually
>>>>>>> punted
>>>>>>> to
>>>>>> elements of an implementation that can’t be usefully deployed.
>>>>>> However,
>>>>>> in
>>>>>> switching ASICs. As I mentioned before, there are many other
>>>>>> components
>>>>>> that
>>> concerns
>>> "easier"
>>> encapsulation
Fabio,

Sorry if it seems like I'm being contentious, but I really would like
to understand the what your concerns are. Your statement was
"Considerations on implementability of an approach is, in my opinion,
a very significant technical concern in selecting a standard.". Yes I
agree, infeasibility in implementation would be a valid technical
objection. But as Matthew as stated in the original email thread
"Please be as concrete and detailed as possible as to your technical
objection.". I would infer that you're objecting that to Geneve and
GUE because they are less implementable in HW than VXLAN-GPE. If that
is indeed your technical objection to those protocols, can you provide
a concrete and detailed explanation why you consider VXLAN-GPE to be
more implementable in HW?

Thanks,
Tom

---

From: Fabio Maino <[email-masked]>
Date: Thu, 4 Aug 2016 17:53:06 -0700

> more implementable in HW?

Tom,
it's more than just infeasibility that should be considered as part of 
the technical evaluation, but also the cost/complexity associated with 
implementations.

Encapsulations are not implemented in a vacuum, but compete with other 
features for resources available in a given platform. VXLAN-GPE was 
designed to be a superset of VXLAN. VXLAN happens to exist as a legacy 
encapsulation and will likely be around for quite some time. An 
implementation of VXLAN-GPE makes a more efficient use of resources on 
platforms that need also to support VXLAN (that to my knowledge are a 
very significant chunk of the implementations out there). This is 
particularly true for fixed-functions ASICs, but I think it stands true 
for other platforms as well when you consider the cost/complexity of 
verification and testing.

Then there's the cost/complexity associated with the transport of 
metadata. I think that a parser for a fixed length portion of a header 
with an ordered set of metadata can be implemented with less 
cost/complexity than a parser for a variable length header where 
metadata can appear in any order. This is particularly evident in 
fixed-function ASICs. IMO, a mandatory to implement encapsulation, 
shouldn't force implementations to deal with variable length unordered 
sets of metadata.

To me those are two important technical aspects to consider, if we have 
to select a single encapsulation mandatory to implement.

Fabio

---

From: Jesse Gross <[email-masked]>
Date: Fri, 5 Aug 2016 14:47:50 +0000

> On Aug 4, 2016, at 5:53 PM, Fabio Maino <[email-masked]> wrote:
> Tom,
> it's more than just infeasibility that should be considered as part of the technical evaluation, but also the cost/complexity associated with implementations.
> 
> Encapsulations are not implemented in a vacuum, but compete with other features for resources available in a given platform. VXLAN-GPE was designed to be a superset of VXLAN. VXLAN happens to exist as a legacy encapsulation and will likely be around for quite some time. An implementation of VXLAN-GPE makes a more efficient use of resources on platforms that need also to support VXLAN (that to my knowledge are a very significant chunk of the implementations out there). This is particularly true for fixed-functions ASICs, but I think it stands true for other platforms as well when you consider the cost/complexity of verification and testing.
> 
> Then there's the cost/complexity associated with the transport of metadata. I think that a parser for a fixed length portion of a header with an ordered set of metadata can be implemented with less cost/complexity than a parser for a variable length header where metadata can appear in any order. This is particularly evident in fixed-function ASICs. IMO, a mandatory to implement encapsulation, shouldn't force implementations to deal with variable length unordered sets of metadata.

The problem is, as you say, this particular encapsulation isn’t implemented in a vacuum. Given the number of variations and extensions that already exist for VXLAN, I think it’s pretty safe to say that things will continue to evolve. Going down the path that you are proposing here would almost certainly mean having several different protocols in the future with different interpretations and possibly lengths. It’s not obvious to me that this is preferable to having a single, extensible protocol from an implementation complexity or testing point of view.

In addition, while you are trying to optimize for one particular implementation and architecture, the long term results of these decisions tend to impact many more platforms than the original one. For example, VXLAN was derived from LISP with the same goals as you are describing here. However, I suspect that many of the platforms that now implement VXLAN never had LISP support in the first place. These implementations must now carry the historical baggage so from a global perspective this did not turn out to be an optimization after all.

---

From: Dino Farinacci <[email-masked]>
Date: Fri, 5 Aug 2016 08:01:18 -0700

> In addition, while you are trying to optimize for one particular implementation and architecture, the long term results of these decisions tend to impact many more platforms than the original one.…

In practice, there really isn’t historical baggage. And if you build something to last 10 years and think it won’t change, having options in the header is good. 

But changing to a different protocol all together and having different options in one protocol IS exactly the same result. THERE ARE CHANGES for the vendor and the deployer.

Dino

---

From: Tom Herbert <[email-masked]>
Date: Fri, 5 Aug 2016 08:38:14 -0700

> Tom,
> it's more than just infeasibility that should be considered as part of the
> technical evaluation, but also the cost/complexity associated with
> implementations.
>
> features for resources available in a given platform. VXLAN-GPE was designed
> to be a superset of VXLAN. VXLAN happens to exist as a legacy encapsulation
> and will likely be around for quite some time. An implementation of
> VXLAN-GPE makes a more efficient use of resources on platforms that need
> also to support VXLAN (that to my knowledge are a very significant chunk of
> the implementations out there). This is particularly true for
> fixed-functions ASICs, but I think it stands true for other platforms as
> well when you consider the cost/complexity of verification and testing.
>

Fabio,

GUE is based on GRE which is one of the world's most mature and
deployed encapsulation protocols. GRE is the only protocol we've found
with variable length headers that is commonly processed by both switch
and NIC hardware. We would have used GRE for our deployment but its
extensibility is too limited.

> Then there's the cost/complexity associated with the transport of metadata.
> I think that a parser for a fixed length portion of a header with an ordered
> set of metadata can be implemented with less cost/complexity than a parser
> for a variable length header where metadata can appear in any order. This is
> particularly evident in fixed-function ASICs. IMO, a mandatory to implement
> encapsulation, shouldn't force implementations to deal with variable length
> unordered sets of metadata.
>
Options in GUE are will ordered, it does not have the combinatoric
complexity associated with TLVs. draft-herbert-gue-extensions shows
the GUE header with all the currently proposed fundament encapsulation
options:

     0                   1                   2                   3
     0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+\
   |        Source port            |      Destination port                  | \
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ UDP
   |           Length              |          Checksum             | /
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+/
   |Ver|C|   Hlen  |  Proto/ctype  |V|SEC|K|F|T|R|   Rsvd Flags    |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                        VNID (optional)                        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                                                               |
   ~                      Security (optional)                      ~
   |                                                               |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                       Checksum (optional)                     |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                                                               |
   +                     Fragmentation (optional)                  +
   |                                                               |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                   Payload transform (optional                 |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                 Remote checksum offload (optional)            |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                                                               |
   ~                   Private fields (optional)                   ~
   |                                                               |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Note that for any combination of options (indicated by set flags)
there is only _one_ possible ordering. So the total number of possible
unique packet formats is 2^N where N is number of option bits. The
number variants of the GUE header is thus 128. To contrast, if these
options were in TLVs the number of possible orderings of the TLVs is
greater than 10,000-- this is the combinatoric nature of TLVs. 128
combinations is small enough that it is feasible to implementing
option parsing with parallel HW techniques (like a TCAM as an
example).

Another important field in the GUE header is Hlen. This provides two
important features:

1) It puts a hard limit on the amount of header that can be used for
options. This is important because a lot of HW implementation can only
load the first N bytes of a packet into their cache for parsing. The
limit also presents the possibility of DOS attack where an attacker
fills up a whole packet with dummy headers like can happen with IPv6
extension headers.
2) The Hlen provides a means for middleboxes to skip over the options
so that it can parse the payload. This means that middleboxes don't
need to understand or parse each option in order to just to DPI on the
payload.

SW implementation is also an important consideration. GUE extensions
are processed without needing a loop and the structure allows random
access. The latter is important if we have to go fishing for specific
options that are needed to processed before others (such as security).
In contrast TLVs and header chains inevitably will require a loop. If
some TLV must be processed early, this in fact may require multiple
passes over the TLVs.

> To me those are two important technical aspects to consider, if we have to
> select a single encapsulation mandatory to implement.
>
As demonstrated, those aspects are well considered in GUE.

Tom

---

From: Jesse Gross <[email-masked]>
Date: Fri, 5 Aug 2016 16:21:46 +0000

> On Aug 5, 2016, at 8:01 AM, Dino Farinacci <[email-masked]> wrote:
> 

I think that if you look at the header layout for VXLAN-GPE (as a further evolution from LISP->VXLAN->VXLAN-GPE) without knowing the historical context, most people would find it a bit odd. In particular, I’ve heard numerous people ask why the ‘I’ bit must be set in VXLAN. These types of issues cause confusion, introduce undefined error states into the protocol, and waste bits in the header. You can argue about whether carrying this legacy forward was a worthwhile tradeoff but it’s clear that there is historical baggage.

> But changing to a different protocol all together and having different options in one protocol IS exactly the same result. THERE ARE CHANGES for the vendor and the deployer.

I don’t agree that the impact from all changes is the same. After all, the stated reason for extending existing header formats is to reduce test and implementation cost. To me, the best way to achieve this is to have a protocol that has a well defined structure that enables incremental extensions while sharing core functionality. In the immediate term this might be a bigger change but over the long term, I believe that it will result in less complexity, and yes, less cost.

---

From: Fabio Maino <[email-masked]>
Date: Fri, 5 Aug 2016 09:27:18 -0700

Actually recomposing the VXLAN/LISP splitting was another design goal of 
VXLAN-GPE. A VXLAN-GPE implementation can share most of its code with a 
LISP implementation, contributing to the reduction of test and 
implementation cost.

Fabio