Subject: Re: [nvo3] New dataplane requirements draft

From: Lizhong Jin <[email-masked]>
Date: Wed, 23 May 2012 22:28:31 +0800

Hi Marc, Nabil,
In draft section 3.3.1.1, it describes the functions of VNID:
"A VNID MUST be included in the overlay encapsulation header on the ingress
NVE when encapsulating a tenant Ethernet frame or IP packet on the overlay
tunnel. The egress NVE uses the VNID to identify the VN context in which
the encapsulated frame should be processed. The VNID can be either a
globally unique identifier (on a per-administrative domain) or a locally
significant identifier. It MUST be easily parsed and processed by the data
path and MAY be distributable by a control-plane or configured via a
management plane."

It does not implicit any format of the VNID. There are potentially two
formats, one is like MPLS lable allocation which is implicit for dataplane
and mapping with control plane information, the other is like VLAN format,
which is explicit for dataplane and could work without control plane. In
http://www.ietf.org/id/draft-kj-nvo3-pion-architecture-00.txt, section 5.3,
it describes this. We should gather some requirement about this, and hope
to see more comments.

Lizhong

---

From: AshwoodsmithPeter <[email-masked]>
Date: Wed, 23 May 2012 15:35:36 +0000

In section 3.3.2.1 LAG and ECMP

"encpsulation headers and/or tunneling methods MUST contain a 'entropy field' or 'entropy label' "

This of course explicitly excludes NVGRE.  I suppose for VXLAN the UDP SourcePort would be considered the entropy field.

Was it the intention to exclude NVGRE?

Peter

---

From: Tina TSOU <[email-masked]>
Date: Wed, 23 May 2012 16:40:12 +0000

Some comments for NVO3 dataplane requirements draft are below.

1.  It is not clear what is the relationship between this document and other drafts that specify NVGRE/VXLAN specifications. If there is none, it must be stated so.

2.  Section 3.2.1 L2 VNI – Regarding  the sentence – “It MUST also be possible to select whether the NVE provides
   optimized multicast trees inside the VNI for individual tenant
   multicast groups or whether the default VNI flooding tree is used.” – Shouldn’t this be the responsibility of Control or management plane i.e to provide the ability to select optimized multicast trees vs default flooding tree ?

3.  Section 3.2.2. L3 VNI -  Regarding the sentence – “The L3 VNI does not require support for Broadcast and Unknown Unicast traffic. The L3 VNI MAY provide support for customer multicast groups”, this is not very clear, although a comment says that the paragraph will be expanded in the future.

4.  3.3.1 NV03 overlay header – how many fields are there, what are their lengths and relative positions? It is better to put a diagram indicating this to make things clear.

5.  Section 3.3.2.1 LAG and ECMP. It is stated “Different flows SHOULD be distributed as evenly as
   possible across multiple underlay network paths. This can be
   achieved by ensuring that some fields used for hashing are randomly
   generated.” – It does not state which fields are used for hashing to calculate ECMP path.

6.  Section 3.4.2.1 – Triangular Routing Issues paragraph not very clear.

a.  What is ELAN – Extended LAN or Stretched LAN ?

b.  It is somewhat indicated that the avoidance of  triangular routing is more of control or management issue. In that case, why is this paragraph relevant and what exactly is the role of dataplane in triangular routing avoidance ?
7.  Section 3.8 - OAM – Are we planning to use any existing OAM standards or extending them to cater to NVO3 needs ? It is not clear.

Tina

On May 23, 2012, at 9:04 AM, "AshwoodsmithPeter" <[email-masked]<mailto:[email-masked]>> wrote:

“encpsulation headers and/or tunneling methods MUST contain a ‘entropy field’ or ‘entropy label’ “

Peter

---

From: "Balus, Florin Stelian (Florin)" <[email-masked]>
Date: Wed, 23 May 2012 12:20:21 -0500

Peter,
We were not trying to exclude any solutions, we tried to really focus on what the requirements should be. Note that the text does not exclude NVGRE anyhow.. We said "...and/or tunnelling methods" and I believe there was a NVGRE thread on this list where the authors discussed how this can be done using tunneling header. There was also some hall discussion in IETF on how this can be done with part of a GRE key.

On the other hand we would like to hear from cloud providers on this: i.e. Is it a MUST or a SHOULD?
Florin

On May 23, 2012, at 8:38 AM, "AshwoodsmithPeter" <[email-masked]<mailto:[email-masked]>> wrote:

Peter
_______________________________________________
nvo3 mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/nvo3

---

From: AshwoodsmithPeter <[email-masked]>
Date: Wed, 23 May 2012 17:55:51 +0000

Hey Florin,

Its just the wording to me implies a requirement for an explicit field or label that contains the entropy. Personally I agree its a good idea (its exactly what we did with 802.1Qbp), but perhaps dropping the 'field'/'label' wording and just indicating that 'the encapsulation MUST result in sufficient entropy to exercise all paths through several LAG/ECMP hops' is more inclusive at the moment?

I don't think anybody will argue that entropy through a few LAG/ECMP hops is a MUST but you never know ;)

Peter

________________________________
From: Balus, Florin Stelian (Florin) [mailto:[email-masked]]
Sent: Wednesday, May 23, 2012 1:20 PM
To: AshwoodsmithPeter
Cc: Lizhong Jin; LASSERRE, MARC (MARC); [email-masked]; [email-masked]
Subject: Re: [nvo3] New dataplane requirements draft

Peter,

Peter

---

From: AshwoodsmithPeter <[email-masked]>
Date: Wed, 23 May 2012 19:16:56 +0000

Florin, I suppose that NVGRE does not have an entropy issue (by your original wording) if it chooses to use an
http://tools.ietf.org/html/rfc6438 flow label but that of course requires IPV6 in the underlay.

Peter

Peter,

Peter

---

From: Carlos Pignataro <[email-masked]>
Date: Wed, 23 May 2012 15:31:08 -0400

> We were not trying to exclude any solutions, we tried to really focus on what the requirements should be. Note that the text does not exclude NVGRE anyhow.. We said "...and/or tunnelling methods" a…

One way is described in http://tools.ietf.org/html/rfc5640 in the context of tunneling for softwires.

Thanks,

-- Carlos.

> On May 23, 2012, at 8:38 AM, "AshwoodsmithPeter" <[email-masked]> wrote:
> 
>> [email-masked]
> [email-masked]

---

From: "Henderickx, Wim (Wim)" <[email-masked]>
Date: Wed, 23 May 2012 21:34:53 +0200

I believe there are multiple ways to solve the hashing with GRE, but if you look at requirements do we need to make this work across existing HW w/o extensions or not. If so the solution set becomes limited.

From: [email-masked] [mailto:[email-masked]] On Behalf Of Carlos Pignataro
Sent: woensdag 23 mei 2012 21:31
To: Balus, Florin Stelian (Florin)
Cc: LASSERRE, MARC (MARC); Lizhong Jin; [email-masked]; [email-masked]; AshwoodsmithPeter
Subject: Re: [nvo3] New dataplane requirements draft

Peter,

Thanks,

-- Carlos.

Peter

---

From: AshwoodsmithPeter <[email-masked]>
Date: Wed, 23 May 2012 19:40:49 +0000

Well that's the key point really. VXLAN puts its entropy in the normal 5 tuple and NVGRE either requires the intermediate hardware to look at the KEY for entropy, or it has to use additional V4 addresses for the outer header, or it has to use V6 and a flow label.

Peter

________________________________
From: Henderickx, Wim (Wim) [mailto:[email-masked]]
Sent: Wednesday, May 23, 2012 3:35 PM
To: Carlos Pignataro; Balus, Florin Stelian (Florin)
Cc: LASSERRE, MARC (MARC); Lizhong Jin; [email-masked]; [email-masked]; AshwoodsmithPeter
Subject: RE: [nvo3] New dataplane requirements draft

Peter,

Thanks,

-- Carlos.

Peter

---

From: "Henderickx, Wim (Wim)" <[email-masked]>
Date: Wed, 23 May 2012 21:50:04 +0200

Indeed, so the question is what we want to achieve short/long term.

From: AshwoodsmithPeter [mailto:[email-masked]]
Sent: woensdag 23 mei 2012 21:41
To: Henderickx, Wim (Wim); Carlos Pignataro; Balus, Florin Stelian (Florin)
Cc: LASSERRE, MARC (MARC); Lizhong Jin; [email-masked]; [email-masked]
Subject: RE: [nvo3] New dataplane requirements draft

Peter

Peter,

Thanks,

-- Carlos.

Peter

---

From: "LASSERRE, MARC (MARC)" <[email-masked]>
Date: Thu, 24 May 2012 10:58:09 +0200

Hi Lizhong,

This is a requirements draft and not a solutions draft where a detailed VNID format needs to be specified.
The intent of the original sentence implies that either a global identifier or a local identifier (such as an MPLS label) can be used.

Are you asking for a specific change in wording - i.e. such as mentioning that a local identifier is typically exchanged via a control plane protocol?
It is pretty implicit and since it is a data plane requirements draft, we did not want to get into control plane related aspects.
If clarification is needed, we will be happy to be a bit more specific.

Marc

________________________________
From: Lizhong Jin [mailto:[email-masked]]
Sent: Wednesday, May 23, 2012 4:29 PM
To: LASSERRE, MARC (MARC); [email-masked]
Cc: [email-masked]
Subject: Re: New dataplane requirements draft

Hi Marc, Nabil,
In draft section 3.3.1.1, it describes the functions of VNID:
"A VNID MUST be included in the overlay encapsulation header on the ingress NVE when encapsulating a tenant Ethernet frame or IP packet on the overlay tunnel. The egress NVE uses the VNID to identify the VN context in which the encapsulated frame should be processed. The VNID can be either a globally unique identifier (on a per-administrative domain) or a locally significant identifier. It MUST be easily parsed and processed by the data path and MAY be distributable by a control-plane or configured via a management plane."

It does not implicit any format of the VNID. There are potentially two formats, one is like MPLS lable allocation which is implicit for dataplane and mapping with control plane information, the other is like VLAN format, which is explicit for dataplane and could work without control plane. In http://www.ietf.org/id/draft-kj-nvo3-pion-architecture-00.txt, section 5.3, it describes this. We should gather some requirement about this, and hope to see more comments.

Lizhong

---

From: "LASSERRE, MARC (MARC)" <[email-masked]>
Date: Thu, 24 May 2012 11:15:34 +0200

Tina,

See my answers inline.

Marc

________________________________
From: Tina TSOU [mailto:[email-masked]]
Sent: Wednesday, May 23, 2012 6:40 PM
To: AshwoodsmithPeter
Cc: Lizhong Jin; LASSERRE, MARC (MARC); [email-masked]; [email-masked]
Subject: Re: [nvo3] New dataplane requirements draft

[ml] This is a generic *data plane requirements* draft. NVGRE, VXLAN, STT are dataplane solutions drafts.

2.  Section 3.2.1 L2 VNI - Regarding  the sentence - "It MUST also be possible to select whether the NVE provides
   optimized multicast trees inside the VNI for individual tenant
   multicast groups or whether the default VNI flooding tree is used." - Shouldn't this be the responsibility of Control or management plane i.e to provide the ability to select optimized multicast trees vs default flooding tree ?

[ml] The decision part (of enabling multicast or using ingress replication) is a control/management aspect. But it is mentioned here in the context of its associated dataplane implications.

3.  Section 3.2.2. L3 VNI -  Regarding the sentence - "The L3 VNI does not require support for Broadcast and Unknown Unicast traffic. The L3 VNI MAY provide support for customer multicast groups", this is not very clear, although a comment says that the paragraph will be expanded in the future.

[ml] There is no broadcast or unknown unicast with IP. Multicast is optional.

4.  3.3.1 NV03 overlay header - how many fields are there, what are their lengths and relative positions? It is better to put a diagram indicating this to make things clear.

[ml] This is specific to dataplane solutions drafts.

5.  Section 3.3.2.1 LAG and ECMP. It is stated "Different flows SHOULD be distributed as evenly as
   possible across multiple underlay network paths. This can be
   achieved by ensuring that some fields used for hashing are randomly
   generated." - It does not state which fields are used for hashing to calculate ECMP path.

[ml] Again, it is up to each dataplane solution to be specific. We have simply provided examples of fields that can be looked up for hashing.

6.  Section 3.4.2.1 - Triangular Routing Issues paragraph not very clear.

a.  What is ELAN - Extended LAN or Stretched LAN ?

[ml] Ethernet LAN

[ml] This text can move to another document once available.

7.  Section 3.8 - OAM - Are we planning to use any existing OAM standards or extending them to cater to NVO3 needs ? It is not clear.

[ml] The gap analysis draft will define whether existing tools are sufficient or if new tools are needed.

Tina

Peter

---

From: "Bitar, Nabil N" <[email-masked]>
Date: Thu, 24 May 2012 08:05:30 -0400

Hi,
We are not specifying the solution, the encapsulation header in this case, but rather that such an identifier must be present in the encapsulation solution that is to be further defined/adopted. There had been additional proposed solutions to what you mentioned.

Thanks,
Nabil

From: Lizhong Jin <[email-masked]<mailto:[email-masked]>>
Date: Wed, 23 May 2012 10:28:31 -0400
To: "LASSERRE, MARC (MARC)" <[email-masked]<mailto:[email-masked]>>, "Bitar, Nabil N" <[email-masked]<mailto:[email-masked]>>
Cc: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: Re: New dataplane requirements draft

Hi Marc, Nabil,

Lizhong

---

From: "Bitar, Nabil N" <[email-masked]>
Date: Thu, 24 May 2012 08:05:47 -0400

Hi Peter,
Agree with Florin that we are not excluding solutions nor specifying a solution. Having said that and just a note, in the  NVGRE case, I believe that the discussion twas hat the reserved field in the GRE header after the TNI could be used for entropy.
Point talen on your comment. However, in your mind based on your comment, would using multiple IP addresses per NVE, effectively multiple tunnels between a pair NVEs and hashing flows to these tunnels be a solution that results in sufficient entropy for instance without having an explicit field in the encapsulation header for entropy? I think that could work but it comes at a disadvantage that can be avoided by having a field that can be use for entropy. We wiill look at the wording in the next rev.

Thanks,
Nabil

From: AshwoodsmithPeter <[email-masked]<mailto:[email-masked]>>
Date: Wed, 23 May 2012 13:55:51 -0400
To: "Balus, Florin Stelian (Florin)" <[email-masked]<mailto:[email-masked]>>
Cc: Lizhong Jin <[email-masked]<mailto:[email-masked]>>, "LASSERRE, MARC (MARC)" <[email-masked]<mailto:[email-masked]>>, "Bitar, Nabil N" <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: RE: [nvo3] New dataplane requirements draft

Hey Florin,

Its just the wording to me implies a requirement for an explicit field or label that contains the entropy. Personally I agree its a good idea (its exactly what we did with 802.1Qbp), but perhaps dropping the ‘field’/’label’ wording and just indicating that ‘the encapsulation MUST result in sufficient entropy to exercise all paths through several LAG/ECMP hops’ is more inclusive at the moment?

I don’t think anybody will argue that entropy through a few LAG/ECMP hops is a MUST but you never know ;)

Peter

Cc: Lizhong Jin; LASSERRE, MARC (MARC); [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>

Peter,

Peter

---

From: Lizhong Jin <[email-masked]>
Date: Thu, 24 May 2012 22:21:11 +0800

Hi Marc, Nabil,
Thank you for the reply. I do not mean specific solution, but hope to see
more specific requirement about VNID, which would be helpful for future
solution. Current requirement is very general to include all possible
solutions.
I could provide two points:
1. do we require any different kinds of control plane to interoperate for
NVO3?
2. do we require dynamic control plane and static configuration to be
interoperate for NVO3?
And yes, a bit related with control plane requirement. If we could answer
the above points, that would be helpful to refine VNID requirement.

Regards
Lizhong

2012/5/24 LASSERRE, MARC (MARC) <[email-masked]>

>  Hi Lizhong,****
>
> ** **
>
> This is a requirements draft and not a solutions draft where a detailed
> VNID format needs to be specified.****
>
> The intent of the original sentence implies that either a global
> identifier or a local identifier (such as an MPLS label) can be used.****
>
> ** **
>
> Are you asking for a specific change in wording – i.e. such as mentioning
> that a local identifier is typically exchanged via a control plane protocol?
> ****
>
> It is pretty implicit and since it is a data plane requirements draft, we
> did not want to get into control plane related aspects. ****
>
> If clarification is needed, we will be happy to be a bit more specific.***
> *
>
> ** **
>
> Marc****
>
> ** **
>  ------------------------------
>
> *From:* Lizhong Jin [mailto:[email-masked]]
> *Sent:* Wednesday, May 23, 2012 4:29 PM
> *To:* LASSERRE, MARC (MARC); [email-masked]
> *Cc:* [email-masked]
> *Subject:* Re: New dataplane requirements draft****
>
> ** **
>
> Hi Marc, Nabil,****
>
> In draft section 3.3.1.1, it describes the functions of VNID:****
>
> "A VNID MUST be included in the overlay encapsulation header on the
> ingress NVE when encapsulating a tenant Ethernet frame or IP packet on the
> overlay tunnel. The egress NVE uses the VNID to identify the VN context in
> which the encapsulated frame should be processed. The VNID can be either a
> management plane."****
> http://www.ietf.org/id/draft-kj-nvo3-pion-architecture-00.txt, section
> 5.3, it describes this. We should gather some requirement about this, and
> hope to see more comments.****
>
>  ****
>
> Lizhong****
>

---

From: AshwoodsmithPeter <[email-masked]>
Date: Thu, 24 May 2012 20:23:51 +0000

Thanks Nabil,

I think in general there is a 'possible' requirement which adds a bit more clarity to the entropy question. You guys do not touch on this in your document but its 'possibly' implied by 3.4 "External NVO3 Connectivity".

Up until now the tunnels are assumed to begin and end inside the DC at the Vswitches , Gateways, etc. but to tunnel out using the same encapsulation they will encounter NAT .. so ....

Possible Requirement:  The NVO3 (where 3=IPV4) encapsulation MUST operates through NAT and ... entropy MUST be preserved.

**if** indeed the above is a requirement I think that it it highlights issues with several of the candidate encapsulations over V4 as follows:

*       I suppose VXLAN would have problems through 1:N NAT because the source port (which is the source of entropy) would get overwritten. So the NAT would polarize VXLAN traffic?

*       Likewise I guess NVGRE would have problems because it does not support 1:N and in 1:1 mode (where the address bits=f(entropy)) it would affect global routing?

V6 is looking better and better!

Cheers

Peter

________________________________
From: [email-masked] [mailto:[email-masked]] On Behalf Of Bitar, Nabil N
Sent: Thursday, May 24, 2012 8:06 AM
To: AshwoodsmithPeter; Balus, Florin Stelian (Florin)
Cc: [email-masked]; LASSERRE, MARC (MARC); Bitar, Nabil N; Lizhong Jin
Subject: Re: [nvo3] New dataplane requirements draft

Hi Peter,

Thanks,
Nabil

Hey Florin,

Peter

Peter,

Peter