Subject: [nvo3] comments on draft-sridharan-virtualization-nvgre-03

From: Lucy yong <[email-masked]>
Date: Tue, 20 Aug 2013 21:48:43 +0000

Hi Authors,

I read this new version and have a major concern on the ECMP support.

*         The new version adds ECMP support section (4.1) and require switches and routers SHOULD provide ECMP on the NVGRE packet using the outer frame fields and entire Key field (32-bit).  This means that the solution not only requires the change on the endpoint devices but also the change on the intermediate switches and routers, which put the high bar for the network. Rare switches and routers today support GRE key based load balance.

*         To support GRE Key based load balancing, the hardware has to check the GRE Key presence and the protocol type to avoid a mis-operation on other GRE usages, which is bad.

*         Using 8 bits in the Key field for flow entropy is not sufficient for all the applications.

*         Directly disclosing VSID to underlying network can be a security concern in some cases.

*         The draft further suggests the interim solution for ECMP support (without any hardware upgrade). It is to assign multiple physical addresses (PA, outer address) to NVGRE end point and use policy controlling the choice of PA to be used. This makes operation complex.

We can have a better solution to support ECMP. Most switches and routers today support 5 tuple based load balance. Five tuple are IP src/dst addr, tcp|udp src/dst ports, and IP protocol type. draft-yong-tsvwg-gre-in-udp-encap-01 proposes the gre-in-udp encapsulation for GRE encapsulated protocols to be tunneled over IP networks where ECMP exists. This solution supports 16 bits flow entropy, does not require any change on intermediate switches and routers, and applies to any GRE encapsulated protocol. It also gives the ingress end point flexibility to generate the flow entropy without explicitly exposing VSID. I highly recommend NVGRE proposal to adopt this method for ECMP support.

Here is the draft. The TSVWG will adopt it as WG draft soon.

http://datatracker.ietf.org/doc/draft-yong-tsvwg-gre-in-udp-encap/

Regards,
Lucy

---

From: Pankaj Garg <[email-masked]>
Date: Thu, 22 Aug 2013 21:18:18 +0000

Hi Lucy,

My comments are inline.

Thanks
Pankaj

From: [email-masked] [mailto:[email-masked]] On Behalf Of Lucy yong
Sent: Tuesday, August 20, 2013 2:49 PM
To: [email-masked]
Cc: [email-masked]; [email-masked]
Subject: [nvo3] comments on draft-sridharan-virtualization-nvgre-03

Hi Authors,

*         The new version adds ECMP support section (4.1) and require switches and routers SHOULD provide ECMP on the NVGRE packet using the outer frame fields and entire Key field (32-bit).  This means that the solution not only requires the change on the endpoint devices but also the change on the intermediate switches and routers, which put the high bar for the network. Rare switches and routers today support GRE key based load balance.
[PG] - This requirement has always been there as without using the key field, there is not enough entropy for ECMP in the network. There are two options for GRE based network virtualization, either use the outer frame fields and the key field or calculate entropy based on the inner packet fields. Many of the newer current generation switches and router silicon supports ECMP hashing for NVGRE packets using outer frame fields and key field. However, the older devices would not be able to do that.

*         Using 8 bits in the Key field for flow entropy is not sufficient for all the applications.
[PG] - The entropy is defined by 4 fields in NVGRE, outer source IP, outer destination IP, VSID and FlowId. The VSID is the only thing that is static for one virtual network. All other fields can change and that can provide additional entropy needed in the large networks.

*         Directly disclosing VSID to underlying network can be a security concern in some cases.
[PG] - In theory, it is possible to use IPsec as the transport to protect the GRE header, however then you lose ECMP etc. I am also not sure why this is a concern since the virtual network ID is disclosed in the network in all network virtualization tunneling proposals out there, correct?

*         The draft further suggests the interim solution for ECMP support (without any hardware upgrade). It is to assign multiple physical addresses (PA, outer address) to NVGRE end point and use policy controlling the choice of PA to be used. This makes operation complex.
[PG] - There is some operational overhead, but I believe it is minimum as one can assign multiple source IP to each NVE to gain additional entropy for ECMP. The tunneling operation can select the source IP based on the hash of inner packet headers.

We can have a better solution to support ECMP. Most switches and routers today support 5 tuple based load balance. Five tuple are IP src/dst addr, tcp|udp src/dst ports, and IP protocol type. draft-yong-tsvwg-gre-in-udp-encap-01 proposes the gre-in-udp encapsulation for GRE encapsulated protocols to be tunneled over IP networks where ECMP exists. This solution supports 16 bits flow entropy, does not require any change on intermediate switches and routers, and applies to any GRE encapsulated protocol. It also gives the ingress end point flexibility to generate the flow entropy without explicitly exposing VSID. I highly recommend NVGRE proposal to adopt this method for ECMP support.
[PG] - Thanks, I would take a look at the proposal, however I treat this as another tunneling protocol in addition to three existing ones i.e. NVGRE, VxLAN and STT. What are the specific advantages of this over other protocols? Also many of current generation NIC hardware support offloads for the existing three protocols. Wouldn't using GRE-IN-UDP break those offloads?

Regards,
Lucy

---

From: Lucy yong <[email-masked]>
Date: Thu, 22 Aug 2013 23:46:28 +0000

Hi Pankaj,

Please see my reply in-line below.

Regards,
Lucy

From: Pankaj Garg [mailto:[email-masked]]
Sent: Thursday, August 22, 2013 4:18 PM
To: Lucy yong; [email-masked]
Cc: [email-masked]; [email-masked]
Subject: RE: comments on draft-sridharan-virtualization-nvgre-03

Hi Lucy,

My comments are inline.

Thanks
Pankaj

Hi Authors,

*         The new version adds ECMP support section (4.1) and require switches and routers SHOULD provide ECMP on the NVGRE packet using the outer frame fields and entire Key field (32-bit).  This means that the solution not only requires the change on the endpoint devices but also the change on the intermediate switches and routers, which put the high bar for the network. Rare switches and routers today support GRE key based load balance.
[PG] - This requirement has always been there as without using the key field, there is not enough entropy for ECMP in the network. There are two options for GRE based network virtualization, either use the outer frame fields and the key field or calculate entropy based on the inner packet fields. Many of the newer current generation switches and router silicon supports ECMP hashing for NVGRE packets using outer frame fields and key field. However, the older devices would not be able to do that.
[Lucy] There is another option beside these two. That is described in the draft mentioned below, which has no change request on intermediate routers/switches. Many older devices are there now. I don't see the reason to replace them for supporting this feature. In addition, both described options create complex logic to existing LB process. Today switches/routers widely support 5 tuple based LB, which we should leverage instead of adding unnecessary complex to it.

*         Using 8 bits in the Key field for flow entropy is not sufficient for all the applications.
[PG] - The entropy is defined by 4 fields in NVGRE, outer source IP, outer destination IP, VSID and FlowId. The VSID is the only thing that is static for one virtual network. All other fields can change and that can provide additional entropy needed in the large networks.
[Lucy] I mean that one virtual network may contain many and many flows, mapping all the flows into 8 bits flow entropy does not give a good granularity, which may potentially cause uneven load balancing issue. Hashing based load balancing is based on statistical. Without a large amount of flow entropy, it will not work well.

*         Directly disclosing VSID to underlying network can be a security concern in some cases.
[PG] - In theory, it is possible to use IPsec as the transport to protect the GRE header, however then you lose ECMP etc. I am also not sure why this is a concern since the virtual network ID is disclosed in the network in all network virtualization tunneling proposals out there, correct?
[Lucy] My point is that the fields that are processed by underlying network (intermediate nodes) should not be the fields belonging to the virtual network. If we want the overlay virtual network is decoupled from the physical infrastructure, we should do that consistently. To let overlay and underlying operate on the same field on packets  is bed idea, IMHO. It creates some dependency and restriction for both networks. For example, what if virtual networks span across several WAN networks, and some WAN operator want to create its own flow entropy by rehashing some fields on the packets? Your two options do not give that flexibility. IMHO, it is important to have flow entropy field on packet header for the underlying network and allow the end point to fill the flow entropy there.

*         The draft further suggests the interim solution for ECMP support (without any hardware upgrade). It is to assign multiple physical addresses (PA, outer address) to NVGRE end point and use policy controlling the choice of PA to be used. This makes operation complex.
[PG] - There is some operational overhead, but I believe it is minimum as one can assign multiple source IP to each NVE to gain additional entropy for ECMP. The tunneling operation can select the source IP based on the hash of inner packet headers.
[Lucy] I don't believe that is simple to manage.

We can have a better solution to support ECMP. Most switches and routers today support 5 tuple based load balance. Five tuple are IP src/dst addr, tcp|udp src/dst ports, and IP protocol type. draft-yong-tsvwg-gre-in-udp-encap-01 proposes the gre-in-udp encapsulation for GRE encapsulated protocols to be tunneled over IP networks where ECMP exists. This solution supports 16 bits flow entropy, does not require any change on intermediate switches and routers, and applies to any GRE encapsulated protocol. It also gives the ingress end point flexibility to generate the flow entropy without explicitly exposing VSID. I highly recommend NVGRE proposal to adopt this method for ECMP support.
[PG] - Thanks, I would take a look at the proposal, however I treat this as another tunneling protocol in addition to three existing ones i.e. NVGRE, VxLAN and STT. What are the specific advantages of this over other protocols? Also many of current generation NIC hardware support offloads for the existing three protocols. Wouldn't using GRE-IN-UDP break those offloads?
[Lucy] If NIC hardware support VxLAN which use UDP port for the flow entropy, it should be easy to support gre-in-udp encaps as well. Will be great  to see chip vendor comment on this. Please take a look at the gre-in-udp encapsulation proposal (below), welcome comment on it.

Regards,
Lucy

Regards,
Lucy

---

From: Pankaj Garg <[email-masked]>
Date: Mon, 26 Aug 2013 18:50:20 +0000

My comments are inline.

From: [email-masked] [mailto:[email-masked]] On Behalf Of Lucy yong
Sent: Thursday, August 22, 2013 4:46 PM
To: Pankaj Garg; [email-masked]
Cc: [email-masked]; [email-masked]
Subject: Re: [nvo3] comments on draft-sridharan-virtualization-nvgre-03

Hi Pankaj,

Regards,
Lucy

Hi Lucy,

My comments are inline.

Thanks
Pankaj

Hi Authors,

*         Using 8 bits in the Key field for flow entropy is not sufficient for all the applications.
[PG] - The entropy is defined by 4 fields in NVGRE, outer source IP, outer destination IP, VSID and FlowId. The VSID is the only thing that is static for one virtual network. All other fields can change and that can provide additional entropy needed in the large networks.
[Lucy] I mean that one virtual network may contain many and many flows, mapping all the flows into 8 bits flow entropy does not give a good granularity, which may potentially cause uneven load balancing issue. Hashing based load balancing is based on statistical. Without a large amount of flow entropy, it will not work well.
[PG] This is incorrect. The entropy is defined by outer source IP, destination IP, VSID and FlowId. In a specific virtual network, only VSID is constant, hence the flow would be mapped based on source IP, destination IP and FlowId. Did you mean to say that between two NVE sending traffic on same VSID, all flows would map based on 8-bit entropy? If yes, then I agree, however different source IP can be used to add additional entropy in such cases if necessary.

*         Directly disclosing VSID to underlying network can be a security concern in some cases.
[PG] - In theory, it is possible to use IPsec as the transport to protect the GRE header, however then you lose ECMP etc. I am also not sure why this is a concern since the virtual network ID is disclosed in the network in all network virtualization tunneling proposals out there, correct?
[Lucy] My point is that the fields that are processed by underlying network (intermediate nodes) should not be the fields belonging to the virtual network. If we want the overlay virtual network is decoupled from the physical infrastructure, we should do that consistently. To let overlay and underlying operate on the same field on packets  is bed idea, IMHO. It creates some dependency and restriction for both networks. For example, what if virtual networks span across several WAN networks, and some WAN operator want to create its own flow entropy by rehashing some fields on the packets? Your two options do not give that flexibility. IMHO, it is important to have flow entropy field on packet header for the underlying network and allow the end point to fill the flow entropy there.
[PG] Yes, this is something not allowed in the spec i.e. modification of GRE key field by transit devices.

*         The draft further suggests the interim solution for ECMP support (without any hardware upgrade). It is to assign multiple physical addresses (PA, outer address) to NVGRE end point and use policy controlling the choice of PA to be used. This makes operation complex.
[PG] - There is some operational overhead, but I believe it is minimum as one can assign multiple source IP to each NVE to gain additional entropy for ECMP. The tunneling operation can select the source IP based on the hash of inner packet headers.
[Lucy] I don't believe that is simple to manage.
[PG] Actually it is not hard, because all you need to do is assign more than one IP to the NVE.  However, it depends upon the operation model. In Hyper-V Network Virtualization, the network virtualization policy is handled by a central controller and hence it is easy to configure the NVE to provide more hashing if needed.

We can have a better solution to support ECMP. Most switches and routers today support 5 tuple based load balance. Five tuple are IP src/dst addr, tcp|udp src/dst ports, and IP protocol type. draft-yong-tsvwg-gre-in-udp-encap-01 proposes the gre-in-udp encapsulation for GRE encapsulated protocols to be tunneled over IP networks where ECMP exists. This solution supports 16 bits flow entropy, does not require any change on intermediate switches and routers, and applies to any GRE encapsulated protocol. It also gives the ingress end point flexibility to generate the flow entropy without explicitly exposing VSID. I highly recommend NVGRE proposal to adopt this method for ECMP support.
[PG] - Thanks, I would take a look at the proposal, however I treat this as another tunneling protocol in addition to three existing ones i.e. NVGRE, VxLAN and STT. What are the specific advantages of this over other protocols? Also many of current generation NIC hardware support offloads for the existing three protocols. Wouldn't using GRE-IN-UDP break those offloads?
[Lucy] If NIC hardware support VxLAN which use UDP port for the flow entropy, it should be easy to support gre-in-udp encaps as well. Will be great  to see chip vendor comment on this. Please take a look at the gre-in-udp encapsulation proposal (below), welcome comment on it.
[PG] Sure, I would take a look at it.

Regards,
Lucy

Regards,
Lucy