Subject: Re: [nvo3] Comments on Draft Geneve

From: Pankaj Garg <[email-masked]>
Date: Sat, 1 Mar 2014 12:22:08 +0000

My comments are inline marked with [PG].

From: nvo3 [mailto:[email-masked]] On Behalf Of Larry Kreeger (kreeger)
Sent: Saturday, March 1, 2014 3:28 AM
To: [email-masked]
Subject: [nvo3] Comments on Draft Geneve

I see that a healthy discussion has broken out around draft-gross-geneve-00 which I see has a slot in the agenda for the NVO3 WG meeting on Monday.  Here are my thoughts.

I will be comparing Geneve to an encapsulation that is near and dear to my heart, VXLAN.  When I do this, I see an encapsulation that is very similar to VXLAN (e.g. uses UDP, uses a 24-bit segment identifier at the same offset).  I see three things that Geneve adds beyond what is available in draft-mahalingam-dutt-dcops-vxlan:

1) The ability to encapsulate any protocol with an Ethertype (not just Ethernet frames), by adding a Protocol Type field.  This is certainly useful, and has already been covered in draft-quinn-vxlan-gpe as a backward compatible extension to VXLAN by using a P bit flag to signal its presence.  The field is even at the same offset as draft-quinn-vxlan-gpe, but is missing the P bit for backwards compatibility.

[PG] The backward compatibility argument is invalid since a frame with P bit set (let me call it VXLAN V2) cannot be processed by the older endpoint, thus having no backward compatibility. Essentially, what you are saying is that one can generate packets in VXLAN V1 for older endpoint and VXLAN V2 for newer endpoints. So the question is, why is VXLAN V2 better than Geneve? In fact, switching on a top level UDP port, provides a cleaner processing pipeline.

2) The addition of an OAM bit to signal that the packet should be processed by the tunnel endpoint and not forwarded to a tenant.  This also seems useful, and seems identical in usage to the (IMO, poorly named) "Router Alert" bit extension to VXLAN covered in (the currently expired) draft-singh-nvo3-vxlan-router-alert.

[PG] Yes, the OAM bit usage is similar. However, this is another extension which is incompatible with older implementation of VXLAN thus breaking backward compatibility.

3) Last, but not least is the addition of a variable length options field, which the draft suggests is used to carry metadata along with the payload.  As mentioned by some others, IMO, the encapsulation transport header is not the right place to define and carry metadata.  Architecturally, metadata should be defined independent of transport so it can be carried inside of whatever transport is desired (e.g. VXLAN, NVGRE, MPLSoGRE, L2TPV3 etc).  One example of an effort to do this is in the Network Service Header draft (draft-quinn-sfc-nsh) being discussed in the SFC WG.  I am guessing that since the Geneve options field is optional, that the metadata it contains is not related to basic network connectivity, but more to providing higher level network services (aka Service Functions).  The Network Service Header contains two separate parts, the service path (used to guide the packets through the service chain) and context (metadata).  I can certainly see the context part of NSH being used to carry metadata even if the service chain is null (all services are fully distributed to the tunnel endpoints).

[PG] The meta-data should be defined by their respective group. Different encapsulation protocols can carry those meta-data in their headers as needed. One clear example of how Geneve is better is that it can carry that meta-data without breaking hardware offloads, whereas VXLAN and NVGRE cannot do that. Btw I want to be clear, Geneve is not defining the meta-data, and it is not tying meta-data to Geneve, it is only defining a general purpose ability to carry meta-data, which is tremendously useful to have in the encapsulation header.

On a side note, I don't believe that the design of NSH is suitable for carrying general purpose meta-data. In fact in its current definition, it is not defining service chaining primitives clearly either, however we can discuss that in SFC forum, and focus the discussion on encapsulation header in this forum.

In short, I don't see anything in Geneve that cannot be accomplished by using the backward compatible extensions to VXLAN proposed in draft-quinn-vxlan-gpe and draft-singh-nvo3-vxlan-router-alert, combined with the addition of NSH.

[PG] Yes, one can put multiple (incompatible) extensions on top of VXLAN, and achieve many things that Geneve is supporting. But at that point, aren't we creating a new encapsulation format altogether? This new protocol with all such extensions would require new hardware, new software, break existing NIC offloads etc. and still carry the legacy baggage with no clear advantage. At that point, I am not sure, why it is better?

When the current NVO3 WG charter was being written, there seemed to be consensus that we have no shortage of encapsulation options, but what was lacking was a standard control plane.  The Geneve draft seems to turn that on its head by saying "There is a clear advantage in settling on a data format: most of the protocols are only superficially different and there is little advantage in duplicating effort.  However, the same cannot be said of control planes, which are diverse in very fundamental ways.  The case for standardization is also less clear given the wide variety in requirements, goals, and deployment scenarios.".  I agree with the first part of this, so why define a completely new, non-backward compatible encapsulation?  I disagree with the second part, since this is clearly the goal of the NVO3 WG.

I see that there is an agenda slot to discuss the Geneve draft, but I'm not clear what the goals are of the authors within the IETF since the draft name does not target it to any particular WG, and it is currently marked as "Informational".  I would suggest that the authors consider extending currently implemented encapsulations rather than starting from scratch, e.g. by moving a few bits around in the first word of the Geneve header, it could be made backward compatible with VXLAN.

Thanks, Larry

---

From: "Bocci, Matthew (Matthew)" <[email-masked]>
Date: Sun, 2 Mar 2014 09:52:22 +0000

Dino

What we said was:

³It's fine to discuss solutions, indeed we need to discuss solutions and
how they meet the requirements and fit with the architecture as input to
the gap analysis draft. However, we are not yet in a position to adopt any
particular solution until we are rechartered to allow this."

Matthew

On 28/02/2014 22:42, "Dino Farinacci" <[email-masked]> wrote:

>So Larry, earlier in the month I asked the chairs if nvo3 was ready to
>talk and explore solutions. The answer was no.
>
>Then why is GENEVE on the agenda chairs?
>
>Dino
>
>> On Feb 28, 2014, at 1:57 PM, "Larry Kreeger (kreeger)"
>><[email-masked]> wrote:
>> 
>> I see that a healthy discussion has broken out around
>>draft-gross-geneve-00 which I see has a slot in the agenda for the NVO3
>>WG meeting on Monday.  Here are my thoughts.
>> 
>> I will be comparing Geneve to an encapsulation that is near and dear to
>>my heart, VXLAN.  When I do this, I see an encapsulation that is very
>>similar to VXLAN (e.g. uses UDP, uses a 24-bit segment identifier at the
>>same offset).  I see three things that Geneve adds beyond what is
>>available in draft-mahalingam-dutt-dcops-vxlan:
>> 
>> 1) The ability to encapsulate any protocol with an Ethertype (not just
>>Ethernet frames), by adding a Protocol Type field.  This is certainly
>>useful, and has already been covered in draft-quinn-vxlan-gpe as a
>>backward compatible extension to VXLAN by using a P bit flag to signal
>>its presence.  The field is even at the same offset as
>>draft-quinn-vxlan-gpe, but is missing the P bit for backwards
>>compatibility.
>> 
>> 2) The addition of an OAM bit to signal that the packet should be
>>processed by the tunnel endpoint and not forwarded to a tenant.  This
>>also seems useful, and seems identical in usage to the (IMO, poorly
>>named) "Router Alert" bit extension to VXLAN covered in (the currently
>>expired) draft-singh-nvo3-vxlan-router-alert.
>> 
>> 3) Last, but not least is the addition of a variable length options
>>field, which the draft suggests is used to carry metadata along with the
>>payload.  As mentioned by some others, IMO, the encapsulation transport
>>header is not the right place to define and carry metadata.
>>Architecturally, metadata should be defined independent of transport so
>>it can be carried inside of whatever transport is desired (e.g. VXLAN,
>>NVGRE, MPLSoGRE, L2TPV3 etc).  One example of an effort to do this is in
>>the Network Service Header draft (draft-quinn-sfc-nsh) being discussed
>>in the SFC WG.  I am guessing that since the Geneve options field is
>>optional, that the metadata it contains is not related to basic network
>>connectivity, but more to providing higher level network services (aka
>>Service Functions).  The Network Service Header contains two separate
>>parts, the service path (used to guide the packets through the service
>>chain) and context (metadata).  I can certainly see the context part of
>>N!
> SH being 
> used to carry metadata even if the service chain is null (all services
>are fully distributed to the tunnel endpoints).
>> 
>> In short, I don't see anything in Geneve that cannot be accomplished by
>>using the backward compatible extensions to VXLAN proposed in
>>draft-quinn-vxlan-gpe and draft-singh-nvo3-vxlan-router-alert, combined
>>with the addition of NSH.
>> 
>> When the current NVO3 WG charter was being written, there seemed to be
>>consensus that we have no shortage of encapsulation options, but what
>>was lacking was a standard control plane.  The Geneve draft seems to
>>turn that on its head by saying "There is a clear advantage in settling
>>on a data format: most of the protocols are only superficially different
>>and there is little advantage in duplicating effort.  However, the same
>>cannot be said of control planes, which are diverse in very fundamental
>>ways.  The case for standardization is also less clear given the wide
>>variety in requirements, goals, and deployment scenarios.".  I agree
>>with the first part of this, so why define a completely new,
>>non-backward compatible encapsulation?  I disagree with the second part,
>>since this is clearly the goal of the NVO3 WG.
>> 
>> I see that there is an agenda slot to discuss the Geneve draft, but I'm
>>not clear what the goals are of the authors within the IETF since the
>>draft name does not target it to any particular WG, and it is currently
>>marked as "Informational".  I would suggest that the authors consider
>>extending currently implemented encapsulations rather than starting from
>>scratch, e.g. by moving a few bits around in the first word of the
>>Geneve header, it could be made backward compatible with VXLAN.
>> 
>> Thanks, Larry
>> _______________________________________________
>> nvo3 mailing list
>> [email-masked]
>> https://www.ietf.org/mailman/listinfo/nvo3
>
>_______________________________________________
>nvo3 mailing list
>[email-masked]
>https://www.ietf.org/mailman/listinfo/nvo3

---

From: "Larry Kreeger (kreeger)" <[email-masked]>
Date: Sun, 2 Mar 2014 15:46:11 +0000

My responses are inline marked with LK>.  - Larry

From: Pankaj Garg <[email-masked]<mailto:[email-masked]>>
Date: Saturday, March 1, 2014 4:22 AM
To: Larry Kreeger <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: RE: Comments on Draft Geneve

From: nvo3 [mailto:[email-masked]] On Behalf Of Larry Kreeger (kreeger)
Sent: Saturday, March 1, 2014 3:28 AM
To: [email-masked]<mailto:[email-masked]>
Subject: [nvo3] Comments on Draft Geneve

[PG] The backward compatibility argument is invalid since a frame with P bit set (let me call it VXLAN V2) cannot be processed by the older endpoint, thus having no backward compatibility.

LK> By backward compatibility, I mean that new implementations of VXLAN (VXLAN V2 as you call it) can understand packets sent by older implementations (VXLAN V1) as well as from new ones.  If older endpoints could understand the future bits, I would call that forward compatibility.

[PG] Essentially, what you are saying is that one can generate packets in VXLAN V1 for older endpoint and VXLAN V2 for newer endpoints. So the question is, why is VXLAN V2 better than Geneve? In fact, switching on a top level UDP port, provides a cleaner processing pipeline.

LK> By enhancing VXLAN, there is no need to get a new UDP port assigned and all the current parsing logic for VXLAN V1 can be applied.

LK> Again, I would call what you are referring to "forward compatibility".

On a side note, I don’t believe that the design of NSH is suitable for carrying general purpose meta-data. In fact in its current definition, it is not defining service chaining primitives clearly either, however we can discuss that in SFC forum, and focus the discussion on encapsulation header in this forum.

[PG] Yes, one can put multiple (incompatible) extensions on top of VXLAN, and achieve many things that Geneve is supporting. But at that point, aren’t we creating a new encapsulation format altogether? This new protocol with all such extensions would require new hardware, new software, break existing NIC offloads etc. and still carry the legacy baggage with no clear advantage. At that point, I am not sure, why it is better?

LK> As I wrote above, extending VXLAN allows the same UDP port to be used and reuse of the existing VXLAN parsing logic.

Thanks, Larry

---

From: Pankaj Garg <[email-masked]>
Date: Sun, 2 Mar 2014 19:06:18 +0000

My responses are inline marked with PG.

From: Larry Kreeger (kreeger) [mailto:[email-masked]]
Sent: Sunday, March 2, 2014 9:16 PM
To: Pankaj Garg; [email-masked]
Subject: Re: Comments on Draft Geneve

LK> By backward compatibility, I mean that new implementations of VXLAN (VXLAN V2 as you call it) can understand packets sent by older implementations (VXLAN V1) as well as from new ones.  If older endpoints could understand the future bits, I would call that forward compatibility.
[PG] My point was that the VXLAN V2 endpoint would have to support generating and understanding VXLAN V1 format packets. Is it much different than an endpoint supporting both Geneve and VXLAN V1?

LK> By enhancing VXLAN, there is no need to get a new UDP port assigned and all the current parsing logic for VXLAN V1 can be applied.
[PG] I am not sure if allocating a new port is the meta issue here. The main issue here seems to be whether new protocol should _require_ support for VXLAN V1 or not. Coming from NVGRE side, the same argument would apply to Geneve where one can say that Geneve should be backward compatible with NVGRE. I feel this might be a slippery slope where a new protocol cannot start with a clean slate.

LK> As I wrote above, extending VXLAN allows the same UDP port to be used and reuse of the existing VXLAN parsing logic.
[PG] The crux of the discussion seems to be, whether Geneve should have a mode that is compatible with VXLAN V1 or not. Even though it might be a slippery slope, I think it is something to think about and debate further.

Thanks, Larry

---

From: Behcet Sarikaya <[email-masked]>
Date: Tue, 11 Mar 2014 09:51:32 -0500

On Fri, Feb 28, 2014 at 3:57 PM, Larry Kreeger (kreeger)
<[email-masked]>wrote:

> draft-gross-geneve-00 which I see has a slot in the agenda for the NVO3 WG
> meeting on Monday.  Here are my thoughts.
>
> same offset).  I see three things that Geneve adds beyond what is available
> in draft-mahalingam-dutt-dcops-vxlan:
>
> useful, and has already been covered in draft-quinn-vxlan-gpe as a backward
> compatible extension to VXLAN by using a P bit flag to signal its presence.
>  The field is even at the same offset as draft-quinn-vxlan-gpe, but is
> missing the P bit for backwards compatibility.
>
> processed by the tunnel endpoint and not forwarded to a tenant.  This also
> seems useful, and seems identical in usage to the (IMO, poorly named)
> "Router Alert" bit extension to VXLAN covered in (the currently
>  Architecturally, metadata should be defined independent of transport so it
> can be carried inside of whatever transport is desired (e.g. VXLAN, NVGRE,
> MPLSoGRE, L2TPV3 etc).  One example of an effort to do this is in the
> Network Service Header draft (draft-quinn-sfc-nsh) being discussed in the
> SFC WG.  I am guessing that since the Geneve options field is optional,
> that the metadata it contains is not related to basic network connectivity,
> but more to providing higher level network services (aka Service
> Functions).  The Network Service Header contains two separate parts, the
> service path (used to guide the packets through the service chain) and
> context (metadata).  I can certainly see the context part of NSH being used
> to carry metadata even if the service chain is null (all services are fully
> distributed to the tunnel endpoints).
>
> using the backward compatible extensions to VXLAN proposed
> in draft-quinn-vxlan-gpe and draft-singh-nvo3-vxlan-router-alert, combined
> consensus that we have no shortage of encapsulation options, but what was
> lacking was a standard control plane.  The Geneve draft seems to turn that
> on its head by saying "There is a clear advantage in settling on a data
> format: most of the protocols are only superficially different and there is
> little advantage in duplicating effort.  However, the same cannot be said
> of control planes, which are diverse in very fundamental ways.  The case
> for standardization is also less clear given the wide variety in
> requirements, goals, and deployment scenarios.".  I agree with the first
> part of this, so why define a completely new, non-backward compatible
> encapsulation?  I disagree with the second part, since this is clearly the
> goal of the NVO3 WG.
>
> not clear what the goals are of the authors within the IETF since the draft
> name does not target it to any particular WG, and it is currently marked as
> "Informational".  I would suggest that the authors consider extending
> currently implemented encapsulations rather than starting from scratch,
> e.g. by moving a few bits around in the first word of the Geneve header, it
> could be made backward compatible with VXLAN.
>
>
+1

I think the above comments are very positive on Geneve but they seem to
have been taken negatively.

Regards,

Behcet