Subject: [nvo3] Taking NVGRE draft to Informational RFC

From: Pankaj Garg <[email-masked]>
Date: Mon, 4 Aug 2014 05:16:07 +0000

Hi All,

As discussed in the NVO3 forum in the past, we intend to take NVGRE to informational RFC status. This would ensure that there is a stable way to build compliant and interoperable NVGRE implementation. It would also allow extensions to NVGRE to be able to refer to a stable document version as opposed to rolling revisions, thus removing the risk of introducing incompatibilities and interoperability issues later on. We are seeking (and would highly appreciate) editorial feedback from NVO3 fellows. Could you please review the latest draft of NVGRE (below) and provide us any editorial feedback on the latest NVGRE draft by 11th August?

Thanks
Pankaj (on behalf of NVGRE co-authors)

---

From: Pankaj Garg <[email-masked]>
Date: Mon, 4 Aug 2014 05:20:10 +0000

Sorry for double post, the latest NVGRE draft is at: http://www.ietf.org/id/draft-sridharan-virtualization-nvgre-05.txt

Thanks
Pankaj

From: nvo3 [mailto:[email-masked]] On Behalf Of Pankaj Garg
Sent: Monday, August 4, 2014 10:46 AM
To: [email-masked]
Cc: [email-masked]
Subject: [nvo3] Taking NVGRE draft to Informational RFC

Hi All,

Thanks

---

From: Tom Herbert <[email-masked]>
Date: Mon, 4 Aug 2014 13:14:38 -0700

On Sun, Aug 3, 2014 at 10:16 PM, Pankaj Garg <[email-masked]> wrote:
> Hi All,
>
>
>
> As discussed in the NVO3 forum in the past, we intend to take NVGRE to
> informational RFC status. This would ensure that there is a stable way to
> build compliant and interoperable NVGRE implementation. It would also allow
> extensions to NVGRE to be able to refer to a stable document version as
> opposed to rolling revisions, thus removing the risk of introducing
> incompatibilities and interoperability issues later on. We are seeking (and
> would highly appreciate) editorial feedback from NVO3 fellows. Could you
> please review the latest draft of NVGRE (below) and provide us any editorial
> feedback on the latest NVGRE draft by 11th August?
>
Hi Pankaj,

Per the draft:

o The C (Checksum Present) and S (Sequence Number Present) bits in
   the GRE header MUST be zero.

This explicitly disallows use of GRE checksum and hence there is no
end to L3 mechanism defined within the protocol to check against
corruption of the nvgre header. As I mentioned before on this list, I
am particularly concerned about the vulnerability of the VSID. I
suspect the intention is that nvgre should only be deployed in
situations where the under lying networks provide error detection
along the whole path (links and in switches). In any case, I would
suggest there should be some discussion of this in the draft.

Thanks,
Tom

> _______________________________________________
> nvo3 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nvo3
>

---

From: Haoweiguo <[email-masked]>
Date: Tue, 5 Aug 2014 02:45:40 +0000

Hi Pankaj,

In this draft, NVGRE encapsulation is clearly defined for east-west direction intra-subnet layer 2 traffic forwarding. And in section 4.6, Cross-subnet, Cross-premise Communication procedures between internal DC and external DC is described, it is for north-south direction traffic forwarding.
For complete solution purpose, I think east-west direction cross-subnet traffic forwarding process had better be discribed also.
Two candidate encapsulations can be used for inter-subnet traffic forwarding:
1. Ethernet in GRE. Destination MAC can be used to distinguish layer 2 or layer 3 traffic. In layer 3 traffic forwarding case, destination MAC is the gateway interface MAC of destination NVE.
2. IP in GRE. Ethernet in GRE encapsulation is only used for intra-subnet layer 2 traffic forwarding. For layer 3 traffic forwarding, IP in GRE encapsulation is used.
In distributed layer 3 gateway scenario, when ingress NVE receives inter-subnet traffic from local TS, if the MAC equals to local gateway interface's MAC, IP forwarding table is looked up.
Then IP in GRE encapsulation is used to transmit the traffic to destination NVE. Destination NVE decapsulates the traffic and looks up local IP forwarding table, then it forwards the traffic to dest TS.

The draft should determine which encapsulation is used to forward inter-subnet layer 3 traffic to faciliate interoperbility among multi vendors.

Thanks

weiguo

________________________________
发件人: Pankaj Garg [[email-masked]]
发送时间: 2014年8月4日 13:20
收件人: Pankaj Garg; [email-masked]
抄送: [email-masked]
主题: Re: [nvo3] Taking NVGRE draft to Informational RFC

Thanks
Pankaj

Hi All,

Thanks

---

From: Haoweiguo <[email-masked]>
Date: Tue, 5 Aug 2014 06:03:43 +0000

Hi Pankaj,

In this draft, NVGRE encapsulation is clearly defined for east-west direction intra-subnet layer 2 traffic forwarding. And in section 4.6, Cross-subnet, Cross-premise Communication procedures between internal DC and external DC is described, it is for north-south direction traffic forwarding.
For complete solution purpose, I think east-west direction cross-subnet traffic forwarding process had better be discribed also.
Two candidate encapsulations can be used for inter-subnet traffic forwarding:
1. Ethernet in GRE.
In this case, both intra-subnet layer 2 traffic and inter-subnet layer 3 traffic use Ethernet in GRE encapsulation. Inner destination MAC can be used to distinguish layer 2 or layer 3 traffic. In layer 2 traffic forwarding case, inner destination MAC is the dest TS's MAC. In layer 3 forwarding case, inner destination MAC is the  gateway interface MAC of destination NVE.

2. IP in GRE.

 In this case, inter-subnet layer 3 traffic use different encapsulation from intra-subnet layer 2 traffic.
Ethernet in GRE encapsulation is only used for intra-subnet layer 2 traffic forwarding. For layer 3 traffic forwarding, IP in GRE encapsulation is used.

In distributed layer 3 gateway scenario, ingress NVE needs to forward both layer 2 and layer 3 traffic at the same time.
When ingress NVE receives inter-subnet traffic from local TS, if the MAC equals to local gateway interface's MAC, IP forwarding table is looked up. Then GRE(Ethernet in GRE or IP in GRE for the layer 3 traffic) encapsulation is used to transmit the traffic to destination NVE. Destination NVE decapsulates the traffic and looks up local IP forwarding table, then it forwards the traffic to dest TS.

The draft should determine which encapsulation is used to forward inter-subnet layer 3 traffic to facilitate interoperbility among multi vendors, i.e., the inter-subnet layer 3 traffic encapsulation format should be Ethernet in GRE or IP in GRE?

Thanks
weiguo

Thanks
Pankaj

Hi All,

Thanks

---

From: Anton Ivanov <[email-masked]>
Date: Tue, 5 Aug 2014 07:13:33 +0100

+1

In the presense of a verified (let's say by hardware) GRE checksum the 
payload checksum can be fed into a generic checksum offload mechanism, 
thus providing an arbitrary checksum offload for the encapsulated traffic.

I do not see the benefits of this restriction. In fact it looks rather 
counterproductive.

So if there are any benefits, they should be discussed in the draft so 
that we can see what is the rationale for it.

A

---

From: "Lizhong Jin" <[email-masked]>
Date: Wed, 6 Aug 2014 16:02:35 +0800

Hi Pankaj, Authors,

The processing of inner VLAN tag for NVGRE and VXLAN is different.

In NVGRE:

   o Inner 802.1Q tag: The inner Ethernet header of NVGRE MUST NOT

   contain 802.1Q Tag. The encapsulating NVE MUST remove any existing

   802.1Q Tag before encapsulation of the frame in NVGRE. A

   decapsulating NVE MUST drop the frame if the inner Ethernet frame

   contains an 802.1Q tag.

In VXLAN:

   Decapsulated VXLAN frames with the inner VLAN tag SHOULD be
   discarded unless configured otherwise.  On the encapsulation side, a
   VTEP SHOULD NOT include an inner VLAN tag on tunnel packets unless
   configured otherwise.  When a VLAN-tagged packet is a candidate for
   VXLAN tunneling, the encapsulating VTEP SHOULD strip the VLAN tag
   unless configured otherwise.

1.       The description in NVGRE draft does not align with that shown in
figure 1. Since inner tag is not allowed, why there is a tag in the figure?

2.       The different mechanism for VXLAN and NVGRE makes confusion to the
implementation for us. IMO, the mechanism in VXLAN is more acceptable. Could
you give some reason why the NVGRE is different with VXLAN? I would
appreciate if the two could have the same tag processing mechanism.

Regards

Lizhong

From: Pankaj Garg [mailto:[email-masked]] 
Sent: 2014年8月4日 13:20
To: Pankaj Garg; [email-masked]
Cc: [email-masked]
Subject: Re: [nvo3] Taking NVGRE draft to Informational RFC

Sorry for double post, the latest NVGRE draft is at:
http://www.ietf.org/id/draft-sridharan-virtualization-nvgre-05.txt

Thanks

Pankaj

Hi All,

Thanks

---

From: "Anton Ivanov (antivano)" <[email-masked]>
Date: Wed, 6 Aug 2014 09:00:25 +0000

While at it,
    Can the Broadcom (specifically) coauthor on the NVGRE draft explain why this is different from other GRE derivatives (SoftGRE) promoted by them where they explicitly mandate VLAN Tags and disallow the use of keys to distinguish between different virtual networks.

Rationale? If any?

A.

On 06/08/14 09:02, Lizhong Jin wrote:
Hi Pankaj, Authors,

1.       The description in NVGRE draft does not align with that shown in figure 1. Since inner tag is not allowed, why there is a tag in the figure?

2.       The different mechanism for VXLAN and NVGRE makes confusion to the implementation for us. IMO, the mechanism in VXLAN is more acceptable. Could you give some reason why the NVGRE is different with VXLAN? I would appreciate if the two could have the same tag processing mechanism.

Regards
Lizhong

From: Pankaj Garg [mailto:[email-masked]]
Sent: 2014年8月4日 13:20
To: Pankaj Garg; [email-masked]<mailto:[email-masked]>
Cc: [email-masked]<mailto:[email-masked]>
Subject: Re: [nvo3] Taking NVGRE draft to Informational RFC

Thanks
Pankaj

From: nvo3 [mailto:[email-masked]] On Behalf Of Pankaj Garg
Sent: Monday, August 4, 2014 10:46 AM
To: [email-masked]<mailto:[email-masked]>
Cc: [email-masked]<mailto:[email-masked]>
Subject: [nvo3] Taking NVGRE draft to Informational RFC

Hi All,

Thanks

_______________________________________________
nvo3 mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/nvo3

---

From: Ian Cox <[email-masked]>
Date: Wed, 6 Aug 2014 17:36:09 +0000

If an intermediate switch/router is performing the encapsulation to NVGRE, and the switch/router is operating in cut-through mode it can start sending the frame out  before receiving the entire frame, this makes it impossible to support GRE checksum in this operating mode.  The argument can be made that if you operate in store and forward mode, you could perform this computation, but the downside to that is increased latency.  This same issue applies to the UDP checksum.  If a checksum can be just on header itself  and not the payload it is more friendly to cut through switching, example IPv4 checksum. 

Ian

-----Original Message-----
From: nvo3 [mailto:[email-masked]] On Behalf Of Anton Ivanov
Sent: Monday, August 04, 2014 11:14 PM
To: [email-masked]
Subject: Re: [nvo3] Taking NVGRE draft to Informational RFC

+1

A

---

From: Tom Herbert <[email-masked]>
Date: Wed, 6 Aug 2014 11:40:13 -0700

> If an intermediate switch/router is performing the encapsulation to NVGRE, and the switch/router is operating in cut-through mode it can start sending the frame out  before receiving the entire fra…
Right, but as you point out that scenario that is only one operating
mode for the protocol. I can certainly envision other modes where the
checksum calculation is feasible and valuable in protecting the VSID.

The UDP issue is similar, however note that in the UDP encapsulation
protocols (e.g. VXLAN) transmitting zero checksum is a SHOULD not a
MUST (also ignoring non-zero UDP checksum at end host is not allowed
by RFC1122).

Per your last point, the checksum is defined by GRE, so I suppose you
could redefine the meaning of the checksum field for the NVGRE version
of the protocol if that is feasible.

Tom

---

From: Pankaj Garg <[email-masked]>
Date: Thu, 7 Aug 2014 04:15:09 +0000

Thanks all for the feedback. I would like to mention that we are looking for "editorial" feedback only to ensure that the document is clear and the text is unambiguous to ensure interoperable implementation can be done. My replies to various comments are below.

[Tom]

"

Per the draft:

This explicitly disallows use of GRE checksum and hence there is no end to L3 mechanism defined within the protocol to check against corruption of the nvgre header. As I mentioned before on this list, I am particularly concerned about the vulnerability of the VSID. I suspect the intention is that nvgre should only be deployed in situations where the under lying networks provide error detection along the whole path (links and in switches). In any case, I would suggest there should be some discussion of this in the draft.

"
[PG] We would clarify this in the deployment requirements. Ethernet CRC can protect against corruption errors. Alternatively IPsec can be used.

[Lizhong]
"
In NVGRE:

2.       The different mechanism for VXLAN and NVGRE makes confusion to the implementation for us. IMO, the mechanism in VXLAN is more acceptable. Could you give some reason why the NVGRE is different with VXLAN? I would appreciate if the two could have the same tag processing mechanism.
"
[PG] I believe recommended VXLAN and allowed NVGRE behavior is same i.e. drop packets with VLAN tag in inner header on decapsulating NVE and strip VLAN tag on encapsulating NVE. Yes, VXLAN makes it optional via configuration to support passing inner VLAN tags and NVGRE does not. We would fix the figure to make sure there is no confusion regarding the same.

[Weiguo]
"

2. IP in GRE.

The draft should determine which encapsulation is used to forward inter-subnet layer 3 traffic to facilitate interoperbility among multi vendors, i.e., the inter-subnet layer 3 traffic encapsulation format should be Ethernet in GRE or IP in GRE?
"
[PG] The encapsulation used for inter-virtualsubnet is same as intra-virtualsubnet encapsulation. We would add a section on inter-virtual subnet forwarding to clarify this.

Thanks
Pankaj

From: Pankaj Garg
Sent: Monday, August 4, 2014 10:50 AM
To: Pankaj Garg; [email-masked]
Cc: [email-masked]
Subject: RE: Taking NVGRE draft to Informational RFC

Thanks
Pankaj

Hi All,

Thanks

---

From: Lucy yong <[email-masked]>
Date: Fri, 8 Aug 2014 17:10:04 +0000

Pankaj,

The description in Section 4.1 is too simple and may cause switch/router implementation issue.

GRE protocol has been used for many years. The Key field may be used differently by different applications. Requiring switch/router to use entire Key field for ECMP works for NVGRE but may break other applications. Because of that, the draft SHOULD make clear that a switch/router MUST check the K bit and Protocol Type value (0x6558) on GRE header before using the Key field for ECMP purpose.

Alternative solution is to use gre-in-udp (draft-ietf-tsv-gre-in-udp), which switches/routers already support today.

Regards,
Lucy

From: nvo3 [mailto:[email-masked]] On Behalf Of Pankaj Garg
Sent: Wednesday, August 06, 2014 11:15 PM
To: [email-masked]
Cc: [email-masked]
Subject: Re: [nvo3] Taking NVGRE draft to Informational RFC

[Tom]

"

Per the draft:

"

[Lizhong]
"
In NVGRE:

[Weiguo]
"

2. IP in GRE.

Thanks
Pankaj

From: Pankaj Garg

Thanks
Pankaj

Hi All,

Thanks

---

From: Pankaj Garg <[email-masked]>
Date: Sat, 9 Aug 2014 03:14:09 +0000

Lucy -

This is NVGRE draft so ECMP recommendation applies to NVGRE only. However, we can certainly make it more obvious that the ECMP recommendation is for NVGRE packets.

Thanks
Pankaj

From: Lucy yong [mailto:[email-masked]]
Sent: Friday, August 8, 2014 10:40 PM
To: Pankaj Garg; [email-masked]
Cc: [email-masked]
Subject: RE: Taking NVGRE draft to Informational RFC

Pankaj,

Regards,
Lucy

[Tom]

"

Per the draft:

"

[Lizhong]
"
In NVGRE:

[Weiguo]
"

2. IP in GRE.

Thanks
Pankaj

From: Pankaj Garg

Thanks
Pankaj

Hi All,

Thanks