Subject: [nvo3] Discussion on encapsulation formats and next steps

From: "Bocci, Matthew (Nokia - GB)" <[email-masked]>
Date: Tue, 4 Oct 2016 09:24:20 +0000

Folks,

Following the lengthy discussion on this list about the pros and cons of the three encapsulation formats, we would like to summarise where the main points of the discussion and to provide some thoughts on next steps.

As a reminder, the question that we asked was: For a given encap, do you have significant technical objections?

Thank you for the lively discussion. We have summarised the key points for each draft as follows:

Geneve
----------
- Can’t be implemented cost-effectively in all use cases because variable length header and order of the TLVs makes is costly (in terms of number of gates) to implement in hardware
- Fork-lift upgrade from widely deployed VXLAN (no backwards compatibility mechanisms)
- Header doesn’t fit into largest commonly available parse buffer (256 bytes in NIC). Cannot justify doubling buffer size unless it is mandatory for hardware to process additional option fields.

GUE
----------
- There were a significant number of objections related to the complexity of implementation in hardware, similar to those noted for Geneve above.
- In addition, there were concerns raised that GUE does not support a sufficient number of extensions due to its reliance on a limited flags field, which is already almost 45% allocated.

VXLAN-GPE
----------
- GPE is not day-1 backwards compatible with VXLAN. Although the frame format is similar, it uses a different UDP port, so would require changes to existing implementations even if the rest of the GPE frame is the same.
- GPE is insufficiently extensible. Numerous extensions and options have been designed for GUE and Geneve. Note that these have not yet been validated by the WG.
- Security e.g. of the VNI has not been addressed by GPE. Although a shim header could be used for security and other extensions, this has not been defined yet and its implications on offloading in NICs are not understood.

Unfortunately, no rough consensus emerged from the list discussion.

The chairs and our AD have also been trying to form a design team to take forward the encapsulation discussion and see if there is potential to design a common encapsulation. However, there has been insufficient interest in this initiative. We would like to hear opinions and confirmation or disagreement on interest in creating a DP encapsulation that addresses the various technical concerns.

For the upcoming Seoul IETF, we propose that we will put aside the discussion of specific encapsulations and focus on control plane and OAM. In particular, the chairs feel there was insufficient discussion of the impact of a software solution that implements some or all of the potential options/extensions allowed by e.g. Geneve on all elements of the NVO3 architecture. We would like the working group to consider more carefully the implications of different encapsulations in real environments consisting of both software and hardware implementations and spanning multiple data centers. For example, OAM functions such as path MTU discovery become challenging with multiple encapsulations along the data path. We would like to encourage solid reviews of the three proposals on the list, particularly how they would work in the general architecture.

With this in mind, we are also considering holding a virtual interim meeting the week of 24th October. More details will follow. 

We would like to start a conversation within the WG about what functionality the WG should focus on and standardize.  What do you think should be easy to do?  What would be incredibly useful?  What, if not done, risks causing harm to the industry?  The start of this discussion of WG direction will occur on the mailing list and in the virtual interim."

Best regards

Matthew and Sam

---

From: Dino Farinacci <[email-masked]>
Date: Tue, 4 Oct 2016 11:24:19 -0700

You said nothing about making a decision. I propose a decision be announced at Seoul IETF.

Dino

> On Oct 4, 2016, at 2:24 AM, Bocci, Matthew (Nokia - GB) <[email-masked]> wrote:
> 
> 
> Folks,
> 
> _______________________________________________
> nvo3 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nvo3

---

From: Alia Atlas <[email-masked]>
Date: Tue, 4 Oct 2016 14:32:21 -0400

Dino,

> You said nothing about making a decision. I propose a decision be

Which decision are you looking for?  Flip a coin & pick one?

Picking an encapsulation despite the technical objections on both sides
will split the market and affect interoperability.
Depending on how much you believe the deployment and key features will be
in software or hardware seriously affects
the arguments.

I have been trying to see if there could be agreement for a design team to
get a solution that takes the objections into
account and produces something.  I have heard from some that they "don't
think it is time for a standard and we should
just provide high quality documentation"; since standards is what the IETF
does, I'm not clear on what the proposed goal
would be otherwise.

By having a solid and thoughtful conversation about the impact of different
encapsulations (and support for different aspects)
connected to the many cases in the NVO3 architecture, I am hoping to get a
more nuanced and thoughtful understanding
of the issues.

Regards,
Alia

> > Following the lengthy discussion on this list about the pros and cons of
> the three encapsulation formats, we would like to summarise where the main
> points of the discussion and to provide some thoughts on next steps.
> >
> > - Can’t be implemented cost-effectively in all use cases because
> variable length header and order of the TLVs makes is costly (in terms of
> number of gates) to implement in hardware
> > - Header doesn’t fit into largest commonly available parse buffer (256
> bytes in NIC). Cannot justify doubling buffer size unless it is mandatory
> for hardware to process additional option fields.
> >
> > GUE
> > ----------
> > - There were a significant number of objections related to the
> complexity of implementation in hardware, similar to those noted for Geneve
> above.
> > - In addition, there were concerns raised that GUE does not support a
> sufficient number of extensions due to its reliance on a limited flags
> field, which is already almost 45% allocated.
> >
> > VXLAN-GPE
> > ----------
> > - GPE is not day-1 backwards compatible with VXLAN. Although the frame
> format is similar, it uses a different UDP port, so would require changes
> to existing implementations even if the rest of the GPE frame is the same.
> > - GPE is insufficiently extensible. Numerous extensions and options have
> been designed for GUE and Geneve. Note that these have not yet been
> validated by the WG.
> > - Security e.g. of the VNI has not been addressed by GPE. Although a
> shim header could be used for security and other extensions, this has not
> been defined yet and its implications on offloading in NICs are not
> understood.
> >
> > The chairs and our AD have also been trying to form a design team to
> take forward the encapsulation discussion and see if there is potential to
> design a common encapsulation. However, there has been insufficient
> interest in this initiative. We would like to hear opinions and
> confirmation or disagreement on interest in creating a DP encapsulation
> that addresses the various technical concerns.
> >
> > For the upcoming Seoul IETF, we propose that we will put aside the
> discussion of specific encapsulations and focus on control plane and OAM.
> In particular, the chairs feel there was insufficient discussion of the
> impact of a software solution that implements some or all of the potential
> options/extensions allowed by e.g. Geneve on all elements of the NVO3
> architecture. We would like the working group to consider more carefully
> the implications of different encapsulations in real environments
> consisting of both software and hardware implementations and spanning
> multiple data centers. For example, OAM functions such as path MTU
> discovery become challenging with multiple encapsulations along the data
> path. We would like to encourage solid reviews of the three proposals on
> the list, particularly how they would work in the general architecture.
> >
> > With this in mind, we are also considering holding a virtual interim
> meeting the week of 24th October. More details will follow.
> >
> > We would like to start a conversation within the WG about what
> functionality the WG should focus on and standardize.  What do you think
> should be easy to do?  What would be incredibly useful?  What, if not done,
> risks causing harm to the industry?  The start of this discussion of WG
> direction will occur on the mailing list and in the virtual interim."
> >
> > Best regards
> >
> > Matthew and Sam
> >
> >
> >
> >
> >
> >

---

From: Joe Touch <[email-masked]>
Date: Tue, 4 Oct 2016 11:38:44 -0700

Hi, all,

On 10/4/2016 11:32 AM, Alia Atlas wrote:
> On Tue, Oct 4, 2016 at 2:24 PM, Dino Farinacci <[email-masked]
> <mailto:[email-masked]>> wrote:
>

My interpretation of the post was "there was no consensus on existing
variants, so we're hoping to figure out what is needed for a new
encapsulation".

Joe

---

From: Lucy yong <[email-masked]>
Date: Tue, 4 Oct 2016 18:45:30 +0000

Maybe these questions help to converge:  does it make a sense to design a common encapsulation protocol at this time and for this space? Do we know how many DC applications using tunnel, and how many are software based and how many are hardware based? We know that two implementations(sw vs. hw) have different objectives, which does not converge now.

Lucy

From: nvo3 [mailto:[email-masked]] On Behalf Of Alia Atlas
Sent: Tuesday, October 04, 2016 1:32 PM
To: Dino Farinacci
Cc: Bocci, Matthew (Nokia - GB); NVO3
Subject: Re: [nvo3] Discussion on encapsulation formats and next steps

Dino,

On Tue, Oct 4, 2016 at 2:24 PM, Dino Farinacci <[email-masked]<mailto:[email-masked]>> wrote:
You said nothing about making a decision. I propose a decision be announced at Seoul IETF.

Picking an encapsulation despite the technical objections on both sides will split the market and affect interoperability.
Depending on how much you believe the deployment and key features will be in software or hardware seriously affects
the arguments.

I have been trying to see if there could be agreement for a design team to get a solution that takes the objections into
account and produces something.  I have heard from some that they "don't think it is time for a standard and we should
just provide high quality documentation"; since standards is what the IETF does, I'm not clear on what the proposed goal
would be otherwise.

By having a solid and thoughtful conversation about the impact of different encapsulations (and support for different aspects)
connected to the many cases in the NVO3 architecture, I am hoping to get a more nuanced and thoughtful understanding
of the issues.

Regards,
Alia

Dino

> On Oct 4, 2016, at 2:24 AM, Bocci, Matthew (Nokia - GB) <[email-masked]<mailto:[email-masked]>> wrote:
>
>
> Folks,
>
> [email-masked]<mailto:[email-masked]>

_______________________________________________
nvo3 mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/nvo3

---

From: Anoop Ghanwani <[email-masked]>
Date: Tue, 4 Oct 2016 11:48:20 -0700

On Tue, Oct 4, 2016 at 2:24 AM, Bocci, Matthew (Nokia - GB) <
[email-masked]> wrote:

> The chairs and our AD have also been trying to form a design team to take
> forward the encapsulation discussion and see if there is potential to
I have little interest in yet another encap.

Anoop

---

From: Linda Dunbar <[email-masked]>
Date: Tue, 4 Oct 2016 18:53:58 +0000

+1.

Linda

From: nvo3 [mailto:[email-masked]] On Behalf Of Anoop Ghanwani
Sent: Tuesday, October 04, 2016 1:48 PM
To: Bocci, Matthew (Nokia - GB)
Cc: NVO3
Subject: Re: [nvo3] Discussion on encapsulation formats and next steps

On Tue, Oct 4, 2016 at 2:24 AM, Bocci, Matthew (Nokia - GB) <[email-masked]<mailto:[email-masked]>> wrote:

Anoop

---

From: Fabio Maino <[email-masked]>
Date: Tue, 4 Oct 2016 11:58:33 -0700

Matthew, Sam,
as it was mentioned in other posts some of the encapsulations you list 
below are implemented and being deployed as we speak.

We have an opportunity to learn, from those real life deployments, what 
are the requirements that needs to be addressed, especially with regard 
to how much extensibility is necessary and what kind of metadata might 
be needed.

I think we should take advantage of that opportunity, and I don't see as 
detrimental that the WG is not going to select or, worst, start afresh 
the design of a single encapsulation.  Control planes will eventually 
"adapt" the data path differences.

Given that we are seeing those encaps in various implementation, it 
would make sense to document them as informational RFC. As an author of 
VXLAN-GPE, I would certainly be happy to work toward that end.

Is there an action that the WG should take to make that happen? Or is it 
left to the initiative of the authors?

Thanks,
Fabio

---

From: Dino Farinacci <[email-masked]>
Date: Tue, 4 Oct 2016 12:29:25 -0700

> Which decision are you looking for?  Flip a coin & pick one?

Pick one. Based on the description text it sounds like one is in favor but the decision has been made explicitly.

> Picking an encapsulation despite the technical objections on both sides will split the market and affect interoperability.

No it won’t. There will be a IETF endorsed encapsulation and then the market will decide which way to go if vendors do more than the IETF endorsed mechanism.

> Depending on how much you believe the deployment and key features will be in software or hardware seriously affects

Yes, that is true. But having said that, THE WG MUST MAKE A DECISION. 

> I have been trying to see if there could be agreement for a design team to get a solution that takes the objections into

It’s too late for that. Discussion had been made by many people and nothing new would come out. Way prolong the inevitable. Make a decision and let’s move on to the control plane.

> just provide high quality documentation"; since standards is what the IETF does, I'm not clear on what the proposed goal

If you don’t decide that VXLAN will remain defacto. And people will think it is the “standard” since its in RFC form. That is how customers think. So if you want the WG to have *some* impact on the industry, decide.

> connected to the many cases in the NVO3 architecture, I am hoping to get a more nuanced and thoughtful understanding

We have done that already. We are beating a dead horse, as to speak.

Dino

---

From: Behcet Sarikaya <[email-masked]>
Date: Tue, 4 Oct 2016 16:59:15 -0500

+1

Behcet

On Tue, Oct 4, 2016 at 1:53 PM, Linda Dunbar <[email-masked]> wrote:
> On Tue, Oct 4, 2016 at 2:24 AM, Bocci, Matthew (Nokia - GB)
> <[email-masked]> wrote:
>
>
>
> forward the encapsulation discussion and see if there is potential to design
> a common encapsulation. However, there has been insufficient interest in
> this initiative. We would like to hear opinions and confirmation or
> disagreement on interest in creating a DP encapsulation that addresses the
> various technical concerns.
>
>
>

---

From: Greg Mirsky <[email-masked]>
Date: Tue, 4 Oct 2016 16:04:33 -0700

Dear Matthew, Sam, et. al,
I think that concentrating on the proposal to "focus on control plane and
OAM" we can achieve practical results despite differences among data plane
encapsulations. Please count me in.

Regards, Greg

On Tue, Oct 4, 2016 at 2:24 AM, Bocci, Matthew (Nokia - GB) <
[email-masked]> wrote:

> - Can’t be implemented cost-effectively in all use cases because variable
> length header and order of the TLVs makes is costly (in terms of number of
> gates) to implement in hardware
> - There were a significant number of objections related to the complexity
> of implementation in hardware, similar to those noted for Geneve above.
> - Security e.g. of the VNI has not been addressed by GPE. Although a shim
> header could be used for security and other extensions, this has not been
> defined yet and its implications on offloading in NICs are not understood.
>

---

From: Sam Aldrin <[email-masked]>
Date: Tue, 4 Oct 2016 20:16:27 -0700

Thanks, Greg, noted.

-sam

---

From: Sam Aldrin <[email-masked]>
Date: Tue, 4 Oct 2016 21:33:35 -0700

Hi Fabio,

<WG chair hat off>
It would be nice to have ONE encap which could ensure that it meets various
requirements of network, which were discussed at length over the course of
NVo3 WG.
As raised by many on the list, there were technical objections raised for
each of the encap.
Making a doc an informational RFC do not solve those objections.
Which means, as a user, would like to see those objections resolved or
addressed.
Deferring to control plane, will only push the complexity of
 interoperability to different plane than solving them.

I, as a user, would like to see DP + CP and OAM, together to meet the
requirements and deployed as solution.
If all three encap proposals provide these options and want to be RFC'ised,
so be it. Not opposed to plurality of encap types.
As of now, having informational RFC, just for DP doesn't solve any, for me.
</WG chair hat off>

cheers
-sam

On Tue, Oct 4, 2016 at 11:58 AM, Fabio Maino <[email-masked]> wrote:

> are the requirements that needs to be addressed, especially with regard to
> how much extensibility is necessary and what kind of metadata might be
> needed.
>
> detrimental that the WG is not going to select or, worst, start afresh the
> design of a single encapsulation.  Control planes will eventually "adapt"
> the data path differences.
>
> Given that we are seeing those encaps in various implementation, it would
> make sense to document them as informational RFC. As an author of

---

From: Sam Aldrin <[email-masked]>
Date: Tue, 4 Oct 2016 21:49:46 -0700

Anoop,

<WG chair hat off>
Couple of questions, if I may ask
1. How do you plan to address technical objections raised?
2. Not interested because it is too late and would rather live with any
deficiencies in the DP proposals?
</WG chair hat off>

-sam
On Tue, Oct 4, 2016 at 11:48 AM, Anoop Ghanwani <[email-masked]>

---

From: Anoop Ghanwani <[email-masked]>
Date: Wed, 5 Oct 2016 09:54:15 -0700

Sam,

My lack of interest in a new encap is because I think it's too late to
converge them.  At this point, there are business issues (as opposed to
technical ones) that would limit the effectiveness of a new encap.  At best
it's a no-op, at worst it creates even more confusion in the market while
the other encaps continue with their deployment.

The best that the IETF can do is at this point is to document these and
make sure the encaps are not breaking something else.

IMO, none of the objections raised are showstoppers.  Any encap can be
modified to do anything we want it to do, with the exception of backwards
compatibility.  The need, efficacy, and the price of backwards
compatibility can be argued, so that advantage is not a slam dunk either.

Anoop

---

From: Dino Farinacci <[email-masked]>
Date: Wed, 5 Oct 2016 09:55:52 -0700

Agree with Anoop.

Dino

> On Oct 5, 2016, at 9:54 AM, Anoop Ghanwani <[email-masked]> wrote:
> 
> Sam,
> 
> My lack of interest in a new encap is because I think it's too late to converge them.  At this point, there are business issues (as opposed to technical ones) that would limit the effectiveness of a new encap.  At best it's a no-op, at worst it creates even more confusion in the market while the other encaps continue with their deployment.
> 
> The best that the IETF can do is at this point is to document these and make sure the encaps are not breaking something else. 
> 
> IMO, none of the objections raised are showstoppers.  Any encap can be modified to do anything we want it to do, with the exception of backwards compatibility.  The need, efficacy, and the price of backwards compatibility can be argued, so that advantage is not a slam dunk either.
> 
> Anoop
> 
> 2. Not interested because it is too late and would rather live with any deficiencies in the DP proposals?
> </WG chair hat off>
> 
> -sam
> On Tue, Oct 4, 2016 at 11:48 AM, Anoop Ghanwani <[email-masked]> wrote:
> On Tue, Oct 4, 2016 at 2:24 AM, Bocci, Matthew (Nokia - GB) <[email-masked]> wrote:
> 
>

---

From: Sam Aldrin <[email-masked]>
Date: Wed, 5 Oct 2016 10:29:15 -0700

Anoop,

As I said in one my earlier emails, if new encap proposals are not
converging on resolving issues, why don't we just live with existing encaps
like VXLAN etc? Why would making these RFC'es is important by standards
body, when it is about business rather than technical ones?

Backward compatibility, extensibility, security, etc., issues are very
important and the degree vary depending on whom you ask, for ex: operator
to vendor, software to hardware. That is whole new discussion and beyond
this thread, but those are the reasons for not reaching rough consensus.
(Ref: mailing list and summary)

I personally do not think WG should just *stamp RFC for drafts because of
business reasons.

-sam

---

From: "Fedyk, Don" <[email-masked]>
Date: Wed, 5 Oct 2016 18:04:15 +0000

Hi Sam

By publishing VXLAN informational RFC with 32 bits of reserved + Reserved Flag bits the IETF invited this situation.  It is inevitable we now need to define the meaning of some of those bits (or even expand the header with TLVs, sigh).  The WG agreeing on the interpretation of some bits in VXLAN can add clarity at this point IMHO.  Also they are just informational/experimental RFCs.

(Agreeing with Anoop)
Cheers,
Don

From: nvo3 [mailto:[email-masked]] On Behalf Of Sam Aldrin
Sent: Wednesday, October 05, 2016 1:29 PM
To: Anoop Ghanwani <[email-masked]>
Cc: Bocci, Matthew (Nokia - GB) <[email-masked]>; NVO3 <[email-masked]>
Subject: Re: [nvo3] Discussion on encapsulation formats and next steps

Anoop,

As I said in one my earlier emails, if new encap proposals are not converging on resolving issues, why don't we just live with existing encaps like VXLAN etc? Why would making these RFC'es is important by standards body, when it is about business rather than technical ones?

Backward compatibility, extensibility, security, etc., issues are very important and the degree vary depending on whom you ask, for ex: operator to vendor, software to hardware. That is whole new discussion and beyond this thread, but those are the reasons for not reaching rough consensus. (Ref: mailing list and summary)

I personally do not think WG should just *stamp RFC for drafts because of business reasons.

-sam

On Wed, Oct 5, 2016 at 9:54 AM, Anoop Ghanwani <[email-masked]<mailto:[email-masked]>> wrote:
Sam,

Anoop

On Tue, Oct 4, 2016 at 9:49 PM, Sam Aldrin <[email-masked]<mailto:[email-masked]>> wrote:
Anoop,

<WG chair hat off>

-sam
On Tue, Oct 4, 2016 at 11:48 AM, Anoop Ghanwani <[email-masked]<mailto:[email-masked]>> wrote:
On Tue, Oct 4, 2016 at 2:24 AM, Bocci, Matthew (Nokia - GB) <[email-masked]<mailto:[email-masked]>> wrote:

Anoop

---

From: Sam Aldrin <[email-masked]>
Date: Wed, 5 Oct 2016 11:11:03 -0700

Don,

VXLAN didn't go through WG route.

Agree that, existing one is broken. The whole effort (with 3 encaps out
there) is to fix what is broken, with a degree of variance.
Summary of issues describes why it is not just about fixing with few bits
:-(

-sam

On Wed, Oct 5, 2016 at 11:04 AM, Fedyk, Don <[email-masked]> wrote:

> Hi Sam
>
>
>
> By publishing VXLAN informational RFC with 32 bits of reserved + Reserved
> Flag bits the IETF invited this situation.  It is inevitable we now need to
> define the meaning of some of those bits (or even expand the header with
> TLVs, sigh).  The WG agreeing on the interpretation of some bits in VXLAN
> can add clarity at this point IMHO.  Also they are just
> informational/experimental RFCs.
>
>
>
> (Agreeing with Anoop)
>
> Cheers,
>
> Don
>
>
>
> *From:* nvo3 [mailto:[email-masked]] *On Behalf Of *Sam Aldrin
> *Sent:* Wednesday, October 05, 2016 1:29 PM
> *To:* Anoop Ghanwani <[email-masked]>
> *Cc:* Bocci, Matthew (Nokia - GB) <[email-masked]>; NVO3 <
> [email-masked]>
> *Subject:* Re: [nvo3] Discussion on encapsulation formats and next steps
>
>
>
> Anoop,
>
>
>

---

From: Joe Touch <[email-masked]>
Date: Wed, 5 Oct 2016 11:45:25 -0700

Hi, all,

On 10/5/2016 10:29 AM, Sam Aldrin wrote:
> converging on resolving issues, why don't we just live with existing
> encaps like VXLAN etc? Why would making these RFC'es is important by
> standards body, when it is about business rather than technical ones? 

My concerns are as follows:

- it seems fine to support multiple encaps, but it's not clear whether
any of the existing encapsulation protocols CAN be "fixed" by OAM
- IMO, the WG shouldn't waste time trying to fix this all

I.e., I would suggest that:

- any encaps that is desired and sufficiently useful should be supported
BUT any deficiencies should be noted, rather than wasting time trying to
"fix" them
- any work on a new encaps needs to be preceded by a requirements doc
that explains what is needed and why; the summary of issues with
existing encaps protocols seems like a list of preferences, not strict
requirements

Joe

---

From: Behcet Sarikaya <[email-masked]>
Date: Wed, 5 Oct 2016 14:02:50 -0500

On Wed, Oct 5, 2016 at 12:29 PM, Sam Aldrin <[email-masked]> wrote:
> important and the degree vary depending on whom you ask, for ex: operator to
> vendor, software to hardware. That is whole new discussion and beyond this
> thread, but those are the reasons for not reaching rough consensus. (Ref:
> mailing list and summary)
>

+1

Behcet
>> compatibility.  The need, efficacy, and the price of backwards compatibility
>> can be argued, so that advantage is not a slam dunk either.
>>
>> Anoop
>>
>>>>> design a common encapsulation. However, there has been insufficient interest
>>>>> in this initiative. We would like to hear opinions and confirmation or

---

From: Greg Mirsky <[email-masked]>
Date: Wed, 5 Oct 2016 12:04:23 -0700

Hi Joe,
my $.02 in-line and tagged GIM>>.

Regards, Greg

> - IMO, the WG shouldn't waste time trying to fix this all
GIM>> I think that common OAM, i.e. Fault Management and Performance
Measurement,  may normalize overlay network operation by presenting single
data model. At the same time, it will support and use any special
capabilities of the particular encapsulation.

> existing encaps protocols seems like a list of preferences, not strict
GIM>> Wholeheartedly agree and would note that that equally applies to work
on the Control Plane and OAM.

---

From: Lucy yong <[email-masked]>
Date: Wed, 5 Oct 2016 19:24:39 +0000

See inline.

From: nvo3 [mailto:[email-masked]] On Behalf Of Sam Aldrin
Sent: Wednesday, October 05, 2016 1:11 PM
To: Fedyk, Don
Cc: Bocci, Matthew (Nokia - GB); Anoop Ghanwani; NVO3
Subject: Re: [nvo3] Discussion on encapsulation formats and next steps

Don,

Agree that, existing one is broken. The whole effort (with 3 encaps out there) is to fix what is broken, with a degree of variance.
Summary of issues describes why it is not just about fixing with few bits :-(
[Lucy] This is not just fixing with few bits. One VXLAN protocol deficiency is the ability to encapsulate IP packets.

Lucy

-sam

On Wed, Oct 5, 2016 at 11:04 AM, Fedyk, Don <[email-masked]<mailto:[email-masked]>> wrote:
Hi Sam

(Agreeing with Anoop)
Cheers,
Don

From: nvo3 [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of Sam Aldrin
Sent: Wednesday, October 05, 2016 1:29 PM
To: Anoop Ghanwani <[email-masked]<mailto:[email-masked]>>
Cc: Bocci, Matthew (Nokia - GB) <[email-masked]<mailto:[email-masked]>>; NVO3 <[email-masked]<mailto:[email-masked]>>
Subject: Re: [nvo3] Discussion on encapsulation formats and next steps

Anoop,

-sam

Anoop

<WG chair hat off>

-sam

Anoop

---

From: Sam Aldrin <[email-masked]>
Date: Wed, 5 Oct 2016 12:48:26 -0700

Joe,

Inline with my comments as individual.

> BUT any deficiencies should be noted, rather than wasting time trying to
We did that for VXLAN, didn't we? Why would it help by adding more encap
types with deficiencies?

> existing encaps protocols seems like a list of preferences, not strict
That is one of the push AD and chairs were trying to do, vis-a-vis, design
team.

-sam

---

From: Anoop Ghanwani <[email-masked]>
Date: Wed, 5 Oct 2016 15:23:06 -0700

Sam,

We can't live with VXLAN because of lack of extensibility--that is one
thing all 3 proposals agree on.

Let's look at how we got here:

The NVO3 charter explicitly mentioned that picking (or working on) a single
encap is a non-goal.  Now the WG has done all the work and arrived at 3
encaps (even 3 is a convergence; there were even more).  It would have been
hard to get convergence to a single encap even back then.  Now, after all
this work, not just in the IETF but actual implementations, I think it's
next to impossible to get convergence.  I would be absolutely thrilled if I
were proven wrong.

Also, just because the NVO3 WG comes up with an encapsulation does not mean
that there won't be NVO3-like implementations using e.g. L2TPv3 or LISP
encapsulations (and those encapsulations are IETF standards).

These are the possibilities I see for the WG and also how I see them
playing out:

1. Publish 3 RFCs -- path of least resistance.  All 3 encaps go through
IETF rigor (security review, etc.).
2. Work on a single encap -- probably a year or two before we have a
standard?  In the meantime the 3 drafts continue to get deployed even
though they expire.  The new encap may or may not see the light of day.
Best case it ends up as 4th encap.  Chances are the earlier encaps will
continue to evolve on their own, each becoming a defacto standard.
3. Do nothing.  Let the drafts expire.  So why did we do all this work?

I picked #1.  If we pick #2 or #3, if the authors want to publish them as
individual submissions, they are free to do so and get RFC numbers anyway.

What are the options that you see and what would be your recommendation for
the way forward?

Anoop

---

From: Tom Herbert <[email-masked]>
Date: Thu, 6 Oct 2016 08:44:30 +0900

> Thank you for the lively discussion. We have summarised the key points for each draft as follows:

Some points of clarification:

The only similarities with Geneve is that they both have variable
length headers and have a header length field. GUE does _not_ use TLVs
and instead uses flag-fields that are well ordered, compact, and
amenable to HW techniques for parsing such as TCAM.

As I understand it the answer to extensibility for VXLAN is to use
NSH. So then if Geneve and GUE with extensions are compared to
VXLAN+NSH then VXLAN+NSH is no less complicated to implement in HW
than Geneve and GUE. Conversely, Geneve and GUE be easily used without
and extensions so in that mode they are no more complicated to
implement in HW than plain VXLAN. I don't think the objection that
VXLAN has less complexity in HW implementation than Geneve or GUE has
much merit.

> - In addition, there were concerns raised that GUE does not support a sufficient number of extensions due to its reliance on a limited flags field, which is already almost 45% allocated.
We can easily extend the the number of flags in GUE the initial 15
bits are allocated (the sixteenth would indicate a field of additional
flags).

Tom

---

From: Joe Touch <[email-masked]>
Date: Wed, 5 Oct 2016 17:01:02 -0700

On 10/5/2016 12:02 PM, Behcet Sarikaya wrote:
>> ...

Agreed, but the WG need not do that. These could be independent
informational docs instead. That's common for de-facto industry
standards already deployed, as long as they don't *interfere* with WG
efforts or undermine the stability of the Internet as a whole.

Joe

---

From: Sam Aldrin <[email-masked]>
Date: Wed, 5 Oct 2016 18:16:55 -0700

On Wed, Oct 5, 2016 at 5:01 PM, Joe Touch <[email-masked]> wrote:

> On Wed, Oct 5, 2016 at 12:29 PM, Sam Aldrin <[email-masked]> <[email-masked]> wrote:
>
> ...> I personally do not think WG should just *stamp RFC for drafts because of> business reasons.>
>
> +1
>
>
> informational docs instead. That's common for de-facto industry standards
> already deployed, as long as they don't *interfere* with WG efforts or
> undermine the stability of the Internet as a whole.
>
> That is what was proposed in the original email about the plans the WG
should embark upon.

-sam

---

From: Sam Aldrin <[email-masked]>
Date: Wed, 5 Oct 2016 19:01:39 -0700

Anoop,

Comments inline, as individual contributor.

On Wed, Oct 5, 2016 at 3:23 PM, Anoop Ghanwani <[email-masked]>
wrote:

> The NVO3 charter explicitly mentioned that picking (or working on) a
> single encap is a non-goal.  Now the WG has done all the work and arrived
> at 3 encaps (even 3 is a convergence; there were even more).  It would have
> been hard to get convergence to a single encap even back then.  Now, after
> all this work, not just in the IETF but actual implementations, I think
> it's next to impossible to get convergence.  I would be absolutely thrilled
> if I were proven wrong.
>
> Also, just because the NVO3 WG comes up with an encapsulation does not
> mean that there won't be NVO3-like implementations using e.g. L2TPv3 or
> LISP encapsulations (and those encapsulations are IETF standards).
>

On the contrary, RFC stamping three proposals doesn't solve those issues
either.

> IETF rigor (security review, etc.).
Why waste WG's time, when technical objections cannot be addressed?

> continue to evolve on their own, each becoming a defacto standard.
Take the scenario, as you suggested, WG reviews the documents and suggests
certain changes to format. Does that create new protocol? Do you/authors
have objections to incorporate those changes? How different is that
situation from the 4th encap as you say?

OR are you saying, WG should just do the nit checking and make them RFC
without making changes to encap format. If so, why bother to go through WG
process?

> 3. Do nothing.  Let the drafts expire.  So why did we do all this work?
Not every draft is bound to become a standard or every effort should be
reflected as RFC.
If there is a need for 3 standard encaps, I am ok, but with a big caveat.
As a user, would like to see technical justification for the existence of 3
encap types, as opposed to business justification you mentioned in previous
email.

> individual submissions, they are free to do so and get RFC numbers anyway.
Sure. That is the option everyone has. VXLAN went through that process and
created problem statement for new encap types.

>
> What are the options that you see and what would be your recommendation
> for the way forward?
>
Plan is as suggested in the original email. Hopefully, we could derive an
agreement and move forward.

-sam

---

From: Anoop Ghanwani <[email-masked]>
Date: Wed, 5 Oct 2016 21:15:32 -0700

Sam,

Sam,

How are the encapsulations supposed to address technical concerns of the
nature "too hard to implement", especially when implementations already
exist?  One can always add extensions to those proposals to address the
other concerns (e.g. VNI not secure, insufficient extensibility, etc.).
Whether those are added now or in the future is immaterial.  If all 3
encaps are willing to address the concern, then we end up with 6 encaps.
Which will we pick then?

I was responding to this statement from the original email.
We would like to hear opinions and confirmation or disagreement on interest
in creating a DP encapsulation that addresses the various technical
concerns.
I still think we are better off with the 3 encapsulations as is, then we
would be if had 4 or 6 encapsulations.
Anoop

On Wed, Oct 5, 2016 at 7:01 PM, Sam Aldrin <[email-masked]> wrote:

> As a user, would like to see technical justification for the existence of
> 3 encap types, as opposed to business justification you mentioned in
> previous email.
>
>
>>>>> On Tue, Oct 4, 2016 at 11:48 AM, Anoop Ghanwani <[email-masked]
>>>>> > wrote:
>>>>>

---

From: Sam Aldrin <[email-masked]>
Date: Wed, 5 Oct 2016 21:46:17 -0700

Anoop,

On Wed, Oct 5, 2016 at 9:15 PM, Anoop Ghanwani <[email-masked]>
wrote:

> exist?
>From various conversations, the sentiment of 'too hard to implement' is not
shared by everyone, that includes silicon vendors supporting different
encap types.
There were discussions over the mailing list about some of those points as
well.

I'd look it from different perspective i.e. don't pick a solution and fit
in the requirements, rather, pick a solution which could meet the
requirements.
Hard, simple, easy etc is a subjective matter and depends on whom you ask.
That is the reason why the request for technical objections were requested.

Having implemented protocols, different versions and changes is a PITA. But
that is the nature of moving things.

>   One can always add extensions to those proposals to address the other
> concerns (e.g. VNI not secure, insufficient extensibility, etc.).  Whether
> those are added now or in the future is immaterial.
>
Immaterial? That is not how networks are deployed/operated, may be in the
lab.

> If all 3 encaps are willing to address the concern, then we end up with 6
> encaps.  Which will we pick then?
>
 RFC7282

-sam

> We would like to hear opinions and confirmation or disagreement on
> interest in creating a DP encapsulation that addresses the various
> technical concerns.
> >>>
>> Take the scenario, as you suggested, WG reviews the documents and
>> suggests certain changes to format. Does that create new protocol? Do
>> you/authors have objections to incorporate those changes? How different is
>> that situation from the 4th encap as you say?
>>
>>> I picked #1.  If we pick #2 or #3, if the authors want to publish them
>>> as individual submissions, they are free to do so and get RFC numbers
>>> anyway.
>>>
>> Sure. That is the option everyone has. VXLAN went through that process
>> and created problem statement for new encap types.
>>
>>>
>>>>> The best that the IETF can do is at this point is to document these
>>>>> and make sure the encaps are not breaking something else.
>>>>>
>>>>>>>> The chairs and our AD have also been trying to form a design team
>>>>>>>> to take forward the encapsulation discussion and see if there is potential
>>>>>>>> to design a common encapsulation. However, there has been insufficient

---

From: Anoop Ghanwani <[email-masked]>
Date: Wed, 5 Oct 2016 21:54:16 -0700

>> encaps.  Which will we pick then?

 So, which of the 3 encaps are we willing to drop based on the last time
the chairs tried to gauge WG consensus? :)

Anoop

---

From: Sam Aldrin <[email-masked]>
Date: Wed, 5 Oct 2016 22:04:10 -0700

Plan is as outlined at the start of the thread.

-sam

On Wed, Oct 5, 2016 at 9:54 PM, Anoop Ghanwani <[email-masked]>
wrote:

>>> If all 3 encaps are willing to address the concern, then we end up with
>>> 6 encaps.  Which will we pick then?
>>>
>>  RFC7282
>>
>

---

From: Lucy yong <[email-masked]>
Date: Thu, 6 Oct 2016 12:20:23 +0000

Sam wrote: As a user, would like to see technical justification for the existence of 3 encap types, as opposed to business justification you mentioned in previous email.

IMO: This is not a valid statement to judge the existence of 3 encaps types. From any single user perspective, he/she only needs one encapsulation type. However, in this area, there are many and many users (or say different DC applications, e.g. cloud, Big data, IoT, etc), which apparently have different technical requirements.

Suggestion: If WG goal is to define one encapsulation protocol to serve all, we need to have a generic protocol structure with flexibility and extensibility to meet different technical requirements; then pick one and work on it. If WG accepts three protocols to meet different technical requirements; WG publishes all three and requires each of them specify the protocol applicability with current specified features.

Lucy

From: nvo3 [mailto:[email-masked]] On Behalf Of Sam Aldrin
Sent: Wednesday, October 05, 2016 9:02 PM
To: Anoop Ghanwani
Cc: Bocci, Matthew (Nokia - GB); NVO3
Subject: Re: [nvo3] Discussion on encapsulation formats and next steps

Anoop,

On Wed, Oct 5, 2016 at 3:23 PM, Anoop Ghanwani <[email-masked]<mailto:[email-masked]>> wrote:
Sam,

We can't live with VXLAN because of lack of extensibility--that is one thing all 3 proposals agree on.

The NVO3 charter explicitly mentioned that picking (or working on) a single encap is a non-goal.  Now the WG has done all the work and arrived at 3 encaps (even 3 is a convergence; there were even more).  It would have been hard to get convergence to a single encap even back then.  Now, after all this work, not just in the IETF but actual implementations, I think it's next to impossible to get convergence.  I would be absolutely thrilled if I were proven wrong.

Also, just because the NVO3 WG comes up with an encapsulation does not mean that there won't be NVO3-like implementations using e.g. L2TPv3 or LISP encapsulations (and those encapsulations are IETF standards).

On the contrary, RFC stamping three proposals doesn't solve those issues either.

These are the possibilities I see for the WG and also how I see them playing out:

1. Publish 3 RFCs -- path of least resistance.  All 3 encaps go through IETF rigor (security review, etc.).
Why waste WG's time, when technical objections cannot be addressed?

2. Work on a single encap -- probably a year or two before we have a standard?  In the meantime the 3 drafts continue to get deployed even though they expire.  The new encap may or may not see the light of day.  Best case it ends up as 4th encap.  Chances are the earlier encaps will continue to evolve on their own, each becoming a defacto standard.
Take the scenario, as you suggested, WG reviews the documents and suggests certain changes to format. Does that create new protocol? Do you/authors have objections to incorporate those changes? How different is that situation from the 4th encap as you say?

OR are you saying, WG should just do the nit checking and make them RFC without making changes to encap format. If so, why bother to go through WG process?

3. Do nothing.  Let the drafts expire.  So why did we do all this work?
Not every draft is bound to become a standard or every effort should be reflected as RFC.
If there is a need for 3 standard encaps, I am ok, but with a big caveat.
As a user, would like to see technical justification for the existence of 3 encap types, as opposed to business justification you mentioned in previous email.

I picked #1.  If we pick #2 or #3, if the authors want to publish them as individual submissions, they are free to do so and get RFC numbers anyway.
Sure. That is the option everyone has. VXLAN went through that process and created problem statement for new encap types.

What are the options that you see and what would be your recommendation for the way forward?
Plan is as suggested in the original email. Hopefully, we could derive an agreement and move forward.

-sam

Anoop

On Wed, Oct 5, 2016 at 10:29 AM, Sam Aldrin <[email-masked]<mailto:[email-masked]>> wrote:
Anoop,

-sam

Anoop

<WG chair hat off>

-sam

Anoop

---

From: Alia Atlas <[email-masked]>
Date: Thu, 6 Oct 2016 10:10:18 -0400

A concern is what is lost by having multiple encapsulation?  Do we require
that a NVE understands the
encapsulation to send to a receiver beyond a physical gateway?  Do we lose
the ability to do traceroute
well (or with enough information) if transit devices don't understand the
particular encapsulation?  What
are the effects if going between multiple data-centers?

What are the implications if there are different limitations on the number
of bytes available for TLVs in
an encapsulation - based upon the implementation?  I've heard that this is
the case for Geneve.

Is there an assumption that transit and gateway nodes don't need to
understand the overlay encapsulation?
Are the same extensions critical for all parts of a path or only for the
NVEs?

I've seen very little thought or discussion given to these issues.  It's
fine to say the control plane can
handle different encapsulations but what does it mean for the architecture?

Does committing to encapsulation A for a data-center mean that data-center
needs a flag-day to go to
using encapsulation B anywhere?  Can data-centers using different
encapsulations successfully communicate
inside a VNI?  How do these answers change if the assumption moves from
virtual machines and a hypervisor
to also handling bare metal servers?

This is the conversation that I would like to see.

Regards,
Alia

On Thu, Oct 6, 2016 at 8:20 AM, Lucy yong <[email-masked]> wrote:

> Sam wrote: As a user, would like to see technical justification for the
> existence of 3 encap types, as opposed to business justification you
> mentioned in previous email.
>
>
>
> IMO: This is not a valid statement to judge the existence of 3 encaps
> types. From any single user perspective, he/she only needs one
> encapsulation type. However, in this area, there are many and many users
> (or say different DC applications, e.g. cloud, Big data, IoT, etc), which
> apparently have different technical requirements.
>
>
>
> Suggestion: If WG goal is to define one encapsulation protocol to serve
> all, we need to have a generic protocol structure with flexibility and
> extensibility to meet different technical requirements; then pick one and
> work on it. If WG accepts three protocols to meet different technical
> requirements; WG publishes all three and requires each of them specify the
> protocol applicability with current specified features.
>
>
>
> Lucy
>
>
>
>
>
> *Sent:* Wednesday, October 05, 2016 9:02 PM
> *To:* Anoop Ghanwani
> *Cc:* Bocci, Matthew (Nokia - GB); NVO3

---

From: Alia Atlas <[email-masked]>
Date: Thu, 6 Oct 2016 10:13:21 -0400

Dino & WG,

Please take a read through RFC3929.  Officially, there needs to be a
consensus call by the WG asking the AD to make a decision.

Regards,
Alia

On Tue, Oct 4, 2016 at 3:29 PM, Dino Farinacci <[email-masked]> wrote:

> No it won’t. There will be a IETF endorsed encapsulation and then the
> market will decide which way to go if vendors do more than the IETF
> endorsed mechanism.
>
> > Depending on how much you believe the deployment and key features will
> be in software or hardware seriously affects
> > the arguments.
>
> > I have been trying to see if there could be agreement for a design team
> to get a solution that takes the objections into
>
> It’s too late for that. Discussion had been made by many people and
> nothing new would come out. Way prolong the inevitable. Make a decision and
> let’s move on to the control plane.
>
> > just provide high quality documentation"; since standards is what the
> IETF does, I'm not clear on what the proposed goal
> > would be otherwise.
>
> If you don’t decide that VXLAN will remain defacto. And people will think
> it is the “standard” since its in RFC form. That is how customers think. So
> if you want the WG to have *some* impact on the industry, decide.
>
> > By having a solid and thoughtful conversation about the impact of
> different encapsulations (and support for different aspects)
> > connected to the many cases in the NVO3 architecture, I am hoping to get
> a more nuanced and thoughtful understanding
> > of the issues.
>
> > > Following the lengthy discussion on this list about the pros and cons
> of the three encapsulation formats, we would like to summarise where the
> main points of the discussion and to provide some thoughts on next steps.
> > >
> > > As a reminder, the question that we asked was: For a given encap, do
> you have significant technical objections?
> > >
> > > - GPE is insufficiently extensible. Numerous extensions and options
> have been designed for GUE and Geneve. Note that these have not yet been
> validated by the WG.

---

From: Behcet Sarikaya <[email-masked]>
Date: Thu, 6 Oct 2016 09:21:22 -0500

#3 Sam already mentioned.
I think we need no action.  As you said, if the authors want to
publish them as individual submissions, they are free to do so and get
RFC numbers anyway.

After VXLAN? Why not use ILA?

Regards,
Behcet

On Wed, Oct 5, 2016 at 5:23 PM, Anoop Ghanwani <[email-masked]> wrote:
> standard?  In the meantime the 3 drafts continue to get deployed even though
> they expire.  The new encap may or may not see the light of day.  Best case
> it ends up as 4th encap.  Chances are the earlier encaps will continue to
> evolve on their own, each becoming a defacto standard.

---

From: "Nagendra Kumar Nainar (naikumar)" <[email-masked]>
Date: Thu, 6 Oct 2016 14:56:50 +0000

Hi Sam,

Please count me in for OAM. It will help to have the Overlay OAM work aligned.

Thanks,
Nagendra

From: nvo3 <[email-masked]<mailto:[email-masked]>> on behalf of Greg Mirsky <[email-masked]<mailto:[email-masked]>>
Date: Tuesday, October 4, 2016 at 7:04 PM
To: "Bocci, Matthew (Nokia - GB)" <[email-masked]<mailto:[email-masked]>>
Cc: NVO3 <[email-masked]<mailto:[email-masked]>>
Subject: Re: [nvo3] Discussion on encapsulation formats and next steps

Dear Matthew, Sam, et. al,
I think that concentrating on the proposal to "focus on control plane and OAM" we can achieve practical results despite differences among data plane encapsulations. Please count me in.

Regards, Greg

Folks,

Geneve
----------
- Can't be implemented cost-effectively in all use cases because variable length header and order of the TLVs makes is costly (in terms of number of gates) to implement in hardware
- Fork-lift upgrade from widely deployed VXLAN (no backwards compatibility mechanisms)
- Header doesn't fit into largest commonly available parse buffer (256 bytes in NIC). Cannot justify doubling buffer size unless it is mandatory for hardware to process additional option fields.

GUE
----------

VXLAN-GPE
----------

Best regards

Matthew and Sam

---

From: Joe Touch <[email-masked]>
Date: Thu, 6 Oct 2016 08:13:21 -0700

On 10/5/2016 6:16 PM, Sam Aldrin wrote:
>
>     These could be independent informational docs instead. That's
>     common for de-facto industry standards already deployed, as long
>     as they don't *interfere* with WG efforts or undermine the
>     stability of the Internet as a whole.
>

The original email did propose that we continue to evaluate existing
encapsulations, but NOT that they be taken out of the WG and published
as individual informational submissions.

Joe

---

From: Sam Aldrin <[email-masked]>
Date: Thu, 6 Oct 2016 08:34:54 -0700

On Thu, Oct 6, 2016 at 8:13 AM, Joe Touch <[email-masked]> wrote:

> These could be independent informational docs instead. That's common for
>> de-facto industry standards already deployed, as long as they don't
>> *interfere* with WG efforts or undermine the stability of the Internet as a
>> whole.
>>
> encapsulations, but NOT that they be taken out of the WG and published as
> individual informational submissions.
>
Correct. Call to go independent route lies with authors, not WG, AFAIK.

-sam

---

From: Joe Touch <[email-masked]>
Date: Thu, 6 Oct 2016 08:57:57 -0700

On 10/6/2016 8:34 AM, Sam Aldrin wrote:
>
> On Thu, Oct 6, 2016 at 8:13 AM, Joe Touch <[email-masked]
> <mailto:[email-masked]>> wrote:
>
>
>     The original email did propose that we continue to evaluate
>     existing encapsulations, but NOT that they be taken out of the WG
>     and published as individual informational submissions.
>
We can't force them to publish as independent, but we do need to decide
that we are not proceeding with the docs as WG items or not.

Joe

---

From: Sam Aldrin <[email-masked]>
Date: Thu, 6 Oct 2016 09:24:03 -0700

> that we are not proceeding with the docs as WG items or not.
I do not think anyone is making or forcing a choice for authors.
As mentioned in the email, there were technical objections, which didn't
help to reach rough consensus by the WG.
Hence, the plan to have discussion leading to Virtual Interim, so we, as
WG, could decide on next steps.

If there are *better* steps which could be undertaken to make progress,
would love to hear them.

-sam

---

From: Anoop Ghanwani <[email-masked]>
Date: Thu, 6 Oct 2016 09:58:19 -0700

Alia,

The obvious way to handle multiple DCs with different encap is to use a
gateway.  The simplest gateway would terminate one encap domain and start
another.  OAM, etc. would work within a domain.  To work across domains,
OAM would have to be run at a higher layer.  This is not too much different
than, e.g., mapping from 802.1ah/PBB to MPLS.

Running multiple encaps within a single DC would be done under supervision
of the NVA.  The NVA would keep track of which NVEs are capable of a given
encap and make sure that a given encap is used only when both the source
and destination NVEs can support it.

It adds complexity but the problems are not insurmountable.

Anoop

---

From: Alia Atlas <[email-masked]>
Date: Thu, 6 Oct 2016 13:29:45 -0400

Anoop,

> than, e.g., mapping from 802.1ah/PBB to MPLS.

So the least common functionality is all that could be assumed?  No
extensions or additional information would be easy to translate.  OAM -
even for things like Path MTU discovery(that matter more if going across a
public Internet between data-centers)  - would not work?  What higher layer
would you see the OAM working at?  Do you have a proposed solution or even
brainstormy ideas?

> and destination NVEs can support it.

Your answer assumes that there is no need for the underlay network to
understand the encapsulation.  It also ignores any issues around multicast.

Can we explore both of them?

> It adds complexity but the problems are not insurmountable.

I hear you - so let's talk about what the problems are and the theoretical
benefits?

>> A concern is what is lost by having multiple encapsulation?  Do we
>> require that a NVE understands the
>> encapsulation to send to a receiver beyond a physical gateway?  Do we
>> lose the ability to do traceroute
>> What are the implications if there are different limitations on the
>> number of bytes available for TLVs in
>> an encapsulation - based upon the implementation?  I've heard that this
>> is the case for Geneve.
>>
>> Does committing to encapsulation A for a data-center mean that
>> data-center needs a flag-day to go to
>>> As a user, would like to see technical justification for the existence
>>> of 3 encap types, as opposed to business justification you mentioned in
>>> previous email.
>>>
>>>
>>>
>>>
>>>

---

From: Alia Atlas <[email-masked]>
Date: Thu, 6 Oct 2016 13:33:58 -0400

> I do not think anyone is making or forcing a choice for authors.

The work is in NVO3.  It is unlikely that work which directly conflicts
with NVO3 could succeed in an Independent Stream submission before the NVO3
WG has published its work.

It isn't clear to me that any of the existing encapsulations would progress
without technical changes through the full IETF review process.

> If there are *better* steps which could be undertaken to make progress,

Absolutely!  I heard quite clearly at the last IETF that there is general
desire for NVO3 to pick a single encapsulation.   While the sentiments on
the list have been more strongly for just publishing everything and
assuming complexity can be handled later, that comes from much less of the
WG than the desire for a single encapsulation.

Regards,
Alia

---

From: Dino Farinacci <[email-masked]>
Date: Thu, 6 Oct 2016 10:50:11 -0700

> Absolutely!  I heard quite clearly at the last IETF that there is general desire for NVO3 to pick a single encapsulation.   While the sentiments on the list have been more strongly for just publishing everything and assuming complexity can be handled later, that comes from much less of the WG than the desire for a single encapsulation.   

Let’s try to reach concensus. Could we agree on one of these 3 options:

(1) Publish all 3 encapsulations as Informational RFCs. This makes the working group look indecisive but at least the vendors can go to market with what they choose with some acknowledgement from IETF.

(2) Pick one encapsulation to be Experimental RFC and the other two Information RFCs. This makes the working group look somewhat decisive but at least allows the vendors to go to market with varying degrees of acknowledgment from IETF.

(3) Close the working group and keep drafts published with current status. Let the solutions working groups work on the control-plane. Because if the working group cannot decide on a simple encapsulation format, there is no way there will be conclusiveness on a control-plane.

This is the only way to get some rough concensus to get out of this stalemate. I vote for (1).

Dino

P.S. A gateway approach is a horrible idea. You will get bad solutions (suboptimal routing) for customers because a standards group couldn’t decide? This doesn’t encourage and promote interoperability and of course pushes the IETF more so into irrelevancy.

---

From: Alia Atlas <[email-masked]>
Date: Thu, 6 Oct 2016 14:38:48 -0400

Dino,

On Thu, Oct 6, 2016 at 1:50 PM, Dino Farinacci <[email-masked]> wrote:

> > Absolutely!  I heard quite clearly at the last IETF that there is
> general desire for NVO3 to pick a single encapsulation.   While the
> sentiments on the list have been more strongly for just publishing
> everything and assuming complexity can be handled later, that comes from
> much less of the WG than the desire for a single encapsulation.
>

Please let Matthew and Sam, as the WG Chairs, determine what calls for
consensus to make and what are reasonable options.

> (1) Publish all 3 encapsulations as Informational RFCs. This makes the
> working group look indecisive but at least the vendors can go to market
> with what they choose with some acknowledgement from IETF.
>

I don't see them going through in the current condition.  Geneve says carry
along arbitrary and unspecified data; that's a security/privacy issue -
maybe resolvable with text.  Geneve says MUST use path MTU discovery, but
header-size can cause issues to get to encapsulated packet header to return
to the VM.

VXLAN-GPE doesn't define what to do with its OAM bit nor is it clear that
is sufficient.  It's likely that more flags and extensions will need to be
defined to handle aspects like security.

> (2) Pick one encapsulation to be Experimental RFC and the other two
> Information RFCs. This makes the working group look somewhat decisive but
> at least allows the vendors to go to market with varying degrees of
> acknowledgment from IETF.
>

Yes, picking an encapsulation would allow documenting as options not picked
(with existing issues described) to go forward.

> (3) Close the working group and keep drafts published with current status.
> Let the solutions working groups work on the control-plane. Because if the
> working group cannot decide on a simple encapsulation format, there is no
> way there will be conclusiveness on a control-plane.
>

I am certainly aware that closing the WG is one of my options.

Regards,
Alia

This is the only way to get some rough concensus to get out of this
> stalemate. I vote for (1).
>
> Dino
>
> P.S. A gateway approach is a horrible idea. You will get bad solutions
> (suboptimal routing) for customers because a standards group couldn’t
> decide? This doesn’t encourage and promote interoperability and of course
> pushes the IETF more so into irrelevancy.
>
>

---

From: Joe Touch <[email-masked]>
Date: Thu, 6 Oct 2016 12:28:24 -0700

On 10/6/2016 11:38 AM, Alia Atlas wrote:
> ....
>  
>
>     (1) Publish all 3 encapsulations as Informational RFCs. This makes
>     the working group look indecisive but at least the vendors can go
>     to market with what they choose with some acknowledgement from IETF.
>
>
> I don't see them going through in the current condition.  Geneve says
> carry along arbitrary and unspecified data; that's a security/privacy
> issue - maybe resolvable with text.  Geneve says MUST use path MTU
> discovery, but header-size can cause issues to get to encapsulated
> packet header to return to the VM.

Most of these proposals are problematic when it comes to MTU and
fragmentation issues. Progressing any of them as WG docs would require
substantial revision to address these issues.

Joe

---

From: Alia Atlas <[email-masked]>
Date: Thu, 6 Oct 2016 16:24:59 -0400

On Thu, Oct 6, 2016 at 3:28 PM, Joe Touch <[email-masked]> wrote:

> carry along arbitrary and unspecified data; that's a security/privacy issue
> - maybe resolvable with text.  Geneve says MUST use path MTU discovery, but

It would be very helpful to have more reviews and discussion of the issues
with each document and thoughts about implications as they fit into a
system larger than a common-encap data-center.  I would welcome more good
technical conversation.

Regards,
Alia

---

From: Tom Herbert <[email-masked]>
Date: Fri, 7 Oct 2016 06:13:42 +0900

On Thu, Oct 6, 2016 at 1:54 AM, Anoop Ghanwani <[email-masked]> wrote:
> The best that the IETF can do is at this point is to document these and make
> sure the encaps are not breaking something else.
>
Lack of security is not a showstopper? No resilience to DDOS attacks
is not a showstopper? No ability to extend the protocol without a
complete hardware forklift is not a showstopper? Maybe this is true in
your datacenter but it certainly is not true in mine. These are
fundamental technical objections with VXLAN, VXLAN-GPE and Geneve.
Neither are they are newly pointed out, for instance I've been asking
for a resolution to the VNI security problem for at least three years
and so far this has not been addressed. If it was as easy as you say
these problems should have been fixed by now.

Tom

---

From: Joe Touch <[email-masked]>
Date: Thu, 6 Oct 2016 14:12:10 -0700

On 10/6/2016 1:24 PM, Alia Atlas wrote:
> It would be very helpful to have more reviews and discussion of the
> issues with each document and thoughts about implications as they fit
> into a system larger than a common-encap data-center.  I would welcome
> more good technical conversation.

The IETF, this WG, and its members have limited cycles. We should not
anticipate full WG review and revision of multiple candidate proposals.

 FWIW, most of what I have to say about IP in X tunnels is in
draft-intarea-tunnels already.

Joe

---

From: Anoop Ghanwani <[email-masked]>
Date: Thu, 6 Oct 2016 15:11:11 -0700

Hi Alia,

Please see inline.

Anoop

> would you see the OAM working at?  Do you have a proposed solution or even

Yes, least common functionality would have to be assumed.  Even if we come
up with encap 4, it will have so many options that most implementations
with implement only a subset of them.  So we will be stuck with the same
issue even with a single encap.  So, the important point here is that
single encap with 10 options doesn't mean all implementations implement all
10 options.

OAM will work in the same way that it works today when translating between
PBB and MPLS/VPLS.  We have OAM running within PBB, OAM running within
MPLS, and OAM running end to end.

> Can we explore both of them?
The underlay network (aside from the gateways) should not need to
understand encapsulation.  There will always be hacky solutions that are
encap aware that claim to do more than the standard underlay, but IMO they
won't be the norm.

Multicast is an interesting one.  First there are several ways to solve
multicast.  Since encap is per source/dest, it won't be an issue for either
head end replication or replication via a multicast service node.  The only
case where it might be a problem with multicast in the underlay and only if
there is a non-intersecting set of encapsulations between the receivers.
There are several solutions like have multiple groups or using a
gateway...of course this is just off the top of my head.  Again,
complexity, but not an insurmountable problem, and I'm probably
trivializing it somewhat.

>>> well (or with enough information) if transit devices don't understand
>>> the particular encapsulation?  What

---

From: Anoop Ghanwani <[email-masked]>
Date: Thu, 6 Oct 2016 16:12:14 -0700

Tom,

I think your note summarizes perfectly why we need to move forward with 3
encapsulations.  Some things matter more to you than they do to me.  If we
start calling for consensus on every one of these requirements, we may end
up with a solution that satisfies nobody.

Anoop

On Thu, Oct 6, 2016 at 2:13 PM, Tom Herbert <[email-masked]> wrote:

> make
> compatibility
> interest

---

From: Tom Herbert <[email-masked]>
Date: Fri, 7 Oct 2016 09:40:51 +0900

Anoop,

These are not requirements but were technical objections raised as
requested by the WG chairs. It is possible that these objections are
not valid, or maybe they can be addressed by updating the drafts with
more protocol or at least clarification. But in any case, I think it
is reasonable to expect that the objections are at least acknowledged
and discussed before moving forward with a protocol-- if we're just
ignoring them then what was the point of taking the time to review the
protocols and raise objections?

Tom

---

From: Lucy yong <[email-masked]>
Date: Fri, 7 Oct 2016 01:35:12 +0000

Repeat my point in early email, there are many DC applications and DC environments, which do not share a common set of technical requirements. What we do?
Personally, I don’t think that having one common encapsulation protocol can reach a consensus now.

Lucy

From: nvo3 [mailto:[email-masked]] On Behalf Of Anoop Ghanwani
Sent: Thursday, October 06, 2016 6:12 PM
To: Tom Herbert
Cc: Sam Aldrin; Bocci, Matthew (Nokia - GB); NVO3
Subject: Re: [nvo3] Discussion on encapsulation formats and next steps

Tom,

I think your note summarizes perfectly why we need to move forward with 3 encapsulations.  Some things matter more to you than they do to me.  If we start calling for consensus on every one of these requirements, we may end up with a solution that satisfies nobody.

Anoop

On Thu, Oct 6, 2016 at 2:13 PM, Tom Herbert <[email-masked]<mailto:[email-masked]>> wrote:

Tom

>>> <[email-masked]<mailto:[email-masked]>> wrote:
>>>>
>>>>
>>>>

---

From: Lucy yong <[email-masked]>
Date: Fri, 7 Oct 2016 01:50:50 +0000

Anoop wrote: Yes, least common functionality would have to be assumed.  Even if we come up with encap 4, it will have so many options that most implementations with implement only a subset of them.  So we will be stuck with the same issue even with a single encap.  So, the important point here is that single encap with 10 options doesn't mean all implementations implement all 10 options.

I do not understand the concern here. IMO: this is the best way to make a common encapsulation protocol to meet all technical requirements, i.e. using options for the requirements are not in common. Then it is up to individual providers to mandate what options are required to be implemented for its application. If a vendor wants to support many applications, it can implement all options; if a vendor only want to support an application, it only implements the options that application requires.  When doing the interworking, it would be per application bases, not per every feature the protocol specifies.

Regards,
Lucy

From: [email-masked] [mailto:[email-masked]] On Behalf Of Anoop Ghanwani
Sent: Thursday, October 06, 2016 5:11 PM
To: Alia Atlas
Cc: Lucy yong; Sam Aldrin; Bocci, Matthew (Nokia - GB); NVO3
Subject: Re: [nvo3] Discussion on encapsulation formats and next steps

Hi Alia,

Please see inline.

Anoop

On Thu, Oct 6, 2016 at 10:29 AM, Alia Atlas <[email-masked]<mailto:[email-masked]>> wrote:
Anoop,

On Thu, Oct 6, 2016 at 12:58 PM, Anoop Ghanwani <[email-masked]<mailto:[email-masked]>> wrote:
Alia,

The obvious way to handle multiple DCs with different encap is to use a gateway.  The simplest gateway would terminate one encap domain and start another.  OAM, etc. would work within a domain.  To work across domains, OAM would have to be run at a higher layer.  This is not too much different than, e.g., mapping from 802.1ah/PBB to MPLS.

So the least common functionality is all that could be assumed?  No extensions or additional information would be easy to translate.  OAM - even for things like Path MTU discovery(that matter more if going across a public Internet between data-centers)  - would not work?  What higher layer would you see the OAM working at?  Do you have a proposed solution or even brainstormy ideas?

Yes, least common functionality would have to be assumed.  Even if we come up with encap 4, it will have so many options that most implementations with implement only a subset of them.  So we will be stuck with the same issue even with a single encap.  So, the important point here is that single encap with 10 options doesn't mean all implementations implement all 10 options.

OAM will work in the same way that it works today when translating between PBB and MPLS/VPLS.  We have OAM running within PBB, OAM running within MPLS, and OAM running end to end.

Running multiple encaps within a single DC would be done under supervision of the NVA.  The NVA would keep track of which NVEs are capable of a given encap and make sure that a given encap is used only when both the source and destination NVEs can support it.

Your answer assumes that there is no need for the underlay network to understand the encapsulation.  It also ignores any issues around multicast.

The underlay network (aside from the gateways) should not need to understand encapsulation.  There will always be hacky solutions that are encap aware that claim to do more than the standard underlay, but IMO they won't be the norm.

Multicast is an interesting one.  First there are several ways to solve multicast.  Since encap is per source/dest, it won't be an issue for either head end replication or replication via a multicast service node.  The only case where it might be a problem with multicast in the underlay and only if there is a non-intersecting set of encapsulations between the receivers.  There are several solutions like have multiple groups or using a gateway...of course this is just off the top of my head.  Again, complexity, but not an insurmountable problem, and I'm probably trivializing it somewhat.

I hear you - so let's talk about what the problems are and the theoretical benefits?

Anoop

On Thu, Oct 6, 2016 at 7:10 AM, Alia Atlas <[email-masked]<mailto:[email-masked]>> wrote:
A concern is what is lost by having multiple encapsulation?  Do we require that a NVE understands the
encapsulation to send to a receiver beyond a physical gateway?  Do we lose the ability to do traceroute
well (or with enough information) if transit devices don't understand the particular encapsulation?  What
are the effects if going between multiple data-centers?

What are the implications if there are different limitations on the number of bytes available for TLVs in
an encapsulation - based upon the implementation?  I've heard that this is the case for Geneve.

Is there an assumption that transit and gateway nodes don't need to understand the overlay encapsulation?
Are the same extensions critical for all parts of a path or only for the NVEs?

I've seen very little thought or discussion given to these issues.  It's fine to say the control plane can
handle different encapsulations but what does it mean for the architecture?

Does committing to encapsulation A for a data-center mean that data-center needs a flag-day to go to
using encapsulation B anywhere?  Can data-centers using different encapsulations successfully communicate
inside a VNI?  How do these answers change if the assumption moves from virtual machines and a hypervisor
to also handling bare metal servers?

Regards,
Alia

On Thu, Oct 6, 2016 at 8:20 AM, Lucy yong <[email-masked]<mailto:[email-masked]>> wrote:
Sam wrote: As a user, would like to see technical justification for the existence of 3 encap types, as opposed to business justification you mentioned in previous email.

Lucy

Anoop,

-sam

Anoop

-sam

Anoop

<WG chair hat off>

-sam

Anoop

---

From: Joe Touch <[email-masked]>
Date: Fri, 7 Oct 2016 07:44:38 -0700

On 10/6/2016 9:58 AM, Anoop Ghanwani wrote:
> The obvious way to handle multiple DCs with different encap is to use
> a gateway.  The simplest gateway would terminate one encap domain and
> start another.  OAM, etc. would work within a domain.  To work across
> domains, OAM would have to be run at a higher layer.  This is not too
> much different than, e.g., mapping from 802.1ah/PBB to MPLS.
>
> Running multiple encaps within a single DC would be done under
> supervision of the NVA.  The NVA would keep track of which NVEs are
> capable of a given encap and make sure that a given encap is used only
> when both the source and destination NVEs can support it.
>

You are assuming that the semantics of different encapsulations are
compatible (MTU, fragmentation, signalling, options), and essentially
describing a translation gateway internetworking model. That model was
replaced by a layered model using one overarching service (IP) for a
reason.

Let's please not reinvent the failed percursors to the Internet.

Joe

---

From: Anoop Ghanwani <[email-masked]>
Date: Fri, 7 Oct 2016 08:13:45 -0700

Joe,

What happens when we have encap 4 and one DC uses 10 options and the other
one uses on one?

Anoop

---

From: Joe Touch <[email-masked]>
Date: Fri, 7 Oct 2016 08:37:22 -0700

If they are "ships in the night" it doesn't matter.

If they interconnect, you either have to know how to translate between these to sets of capabilities or fail. Using the common subset of the two means turning off some of the features, which avoids translation failures only if it can be negotiated (and both sides agree).

This is a rehash of how internetworking already deals with heterogeneity. 

Joe

> On Oct 7, 2016, at 8:13 AM, Anoop Ghanwani <[email-masked]> wrote:
> 
> Joe,
> 
> What happens when we have encap 4 and one DC uses 10 options and the other one uses on one?
> 
> Anoop
>

---

From: Behcet Sarikaya <[email-masked]>
Date: Fri, 7 Oct 2016 11:23:14 -0500

Anoop wrote on Oct. 4

Now Anoop is arguing for another encap or encaps.
I don't understand why this change of heart happened?
Of course we have freedom to change mind.

Behcet

On Thu, Oct 6, 2016 at 5:11 PM, Anoop Ghanwani <[email-masked]> wrote:
>>> another.  OAM, etc. would work within a domain.  To work across domains, OAM
>>> would have to be run at a higher layer.  This is not too much different
>> extensions or additional information would be easy to translate.  OAM - even
>> for things like Path MTU discovery(that matter more if going across a public
>> Internet between data-centers)  - would not work?  What higher layer would
>> you see the OAM working at?  Do you have a proposed solution or even
>> brainstormy ideas?
>
>
> up with encap 4, it will have so many options that most implementations with
> implement only a subset of them.  So we will be stuck with the same issue
> even with a single encap.  So, the important point here is that single encap
> with 10 options doesn't mean all implementations implement all 10 options.
>
> PBB and MPLS/VPLS.  We have OAM running within PBB, OAM running within MPLS,
> and OAM running end to end.
>
>>
>>
>>> supervision of the NVA.  The NVA would keep track of which NVEs are capable
>>> of a given encap and make sure that a given encap is used only when both the
>>> source and destination NVEs can support it.
>>
>>
> The underlay network (aside from the gateways) should not need to understand
> encapsulation.  There will always be hacky solutions that are encap aware
> that claim to do more than the standard underlay, but IMO they won't be the
> norm.
>
> gateway...of course this is just off the top of my head.  Again, complexity,
> but not an insurmountable problem, and I'm probably trivializing it
> somewhat.
>
>>
>>
>>>
>>>>> types. From any single user perspective, he/she only needs one encapsulation
>>>>> type. However, in this area, there are many and many users (or say different
>>>>> DC applications, e.g. cloud, Big data, IoT, etc), which apparently have
>>>>> different technical requirements.
>>>>>
>>>>>
>>>>>
>>>>> single encap is a non-goal.  Now the WG has done all the work and arrived at
>>>>> 3 encaps (even 3 is a convergence; there were even more).  It would have
>>>>> all this work, not just in the IETF but actual implementations, I think it's
>>>>> mean that there won't be NVO3-like implementations using e.g. L2TPv3 or LISP

---

From: Anoop Ghanwani <[email-masked]>
Date: Fri, 7 Oct 2016 10:44:48 -0700

Huh?

On Fri, Oct 7, 2016 at 9:23 AM, Behcet Sarikaya <[email-masked]>
wrote:

> even
> with
> capable
> understand
> complexity,
> encapsulation
> >>>>> Plan is as suggested in the original email. Hopefully, we could
> derive
> >>>>> an agreement and move forward.
> >>>>>
> >>>>>
> >>>>>
> >>>>> -sam
> >>>>>
> >>>>>
> >>>>>
> >>>>> Anoop
> >>>>>
> >>>>>
> >>>>>
> encaps
> >>>>> I personally do not think WG should just *stamp RFC for drafts
> because
> >>>>> of business reasons.
> >>>>>
> >>>>>
> >>>>>
> >>>>> -sam
> >>>>>
> >>>>>
> >>>>>
> and

---

From: Dino Farinacci <[email-masked]>
Date: Fri, 7 Oct 2016 11:06:17 -0700

“The problem with standards is that there are many standards”. I forgot who coined the phrase but its an old phrase.

Sad but true,
Dino

> On Oct 7, 2016, at 9:23 AM, Behcet Sarikaya <[email-masked]> wrote:
> 
> Anoop wrote on Oct. 4
>

---

From: "Kiran.Makhijani" <[email-masked]>
Date: Fri, 7 Oct 2016 22:29:19 +0000

Anoop, Joe
You are both making interesting points about capability negotiation, leading to this question for the group –Is it even feasible to proceed with OAM or control plane activity without assuming (or referring to) any base encap?
Thanks
Kiran

From: nvo3 [mailto:[email-masked]] On Behalf Of Joe Touch
Sent: Friday, October 07, 2016 8:37 AM
To: Anoop Ghanwani <[email-masked]>
Cc: Sam Aldrin <[email-masked]>; Bocci, Matthew (Nokia - GB) <[email-masked]>; NVO3 <[email-masked]>; Lucy yong <[email-masked]>; Alia Atlas <[email-masked]>
Subject: Re: [nvo3] Discussion on encapsulation formats and next steps

Joe

On Oct 7, 2016, at 8:13 AM, Anoop Ghanwani <[email-masked]<mailto:[email-masked]>> wrote:
Joe,

Anoop

On Fri, Oct 7, 2016 at 7:44 AM, Joe Touch <[email-masked]<mailto:[email-masked]>> wrote:

Joe

---

From: Joe Touch <[email-masked]>
Date: Fri, 7 Oct 2016 16:21:08 -0700

Hi, Kiran,

I would assume that it is possible to support NVO3 deployment and
management if each overlay uses only one encapsulation protocol. The
issue is deploying a single overlay using multiple protocols - that's
where things get difficult, if not impossible. It is more challenging to
deal with heterogeneity - the OAM system would need to discover
capabilities and deploy overlays only where there was a common subset,
or match the capabilities available to the user/operator provided
constraints.

That's basically how we addressed this issue in the X-Bone overlay
deployment system. It supported different encapsulations (IPv4, IPv6,
IPv4 with IPsec, IPv6 with IPsec), but each overlay system deployed used
only one encapsulation throughout. Users/operators indicated
requirements that mapped directly to encapsulation configurations and we
had a secure capability discovery system where "invitations" asked for
particular capabilities and only nodes willing to participate responded.

I'm not sure how much of this maps to a more modern OAM system, but it
seems to me that this is an example that heterogeneous encaps protocols
can be supported usefully without too much difficulty, - again, as long
as each overlay is homogenous.

Joe

On 10/7/2016 3:29 PM, Kiran.Makhijani wrote:
>
> Anoop, Joe
>
> You are both making interesting points about capability negotiation,
> leading to this question for the group –Is it even feasible to proceed
> with OAM or control plane activity without assuming (or referring to)
> any base encap?
>
> Thanks
>
> Kiran
>
>  
>
> *From:*nvo3 [mailto:[email-masked]] *On Behalf Of *Joe Touch
> *Sent:* Friday, October 07, 2016 8:37 AM
> *Cc:* Sam Aldrin <[email-masked]>; Bocci, Matthew (Nokia - GB)
> <[email-masked]>; NVO3 <[email-masked]>; Lucy yong
> <[email-masked]>; Alia Atlas <[email-masked]>
> If they interconnect, you either have to know how to translate between
> these to sets of capabilities or fail. Using the common subset of the
> two means turning off some of the features, which avoids translation
> failures only if it can be negotiated (and both sides agree).
>
>  
>

---

From: Greg Mirsky <[email-masked]>
Date: Sat, 8 Oct 2016 10:43:46 -0700

Hi Joe,
I think that he task of OAM interworking between two NVO3 domains that use
different dataplane encapsulations would be much simpler if their
respective OAM are compliant to the same comprehensive set of requirements.
That is the idea and the goal the Overlay OAM Design Team worked towards -
common requirements that result in common OAM functionality across
different overlay networks. And the next step we've considered - common
active OAM for both fault management and performance measurement tasks. We
are preparing update to our Common OOAM Header draft to propose format that
will support common way of embedding OAM control message into data packets.

Regards, Greg

On Fri, Oct 7, 2016 at 4:21 PM, Joe Touch <[email-masked]> wrote:

> management if each overlay uses only one encapsulation protocol. The issue
> is deploying a single overlay using multiple protocols - that's where
> things get difficult, if not impossible. It is more challenging to deal
> with heterogeneity - the OAM system would need to discover capabilities and
> deploy overlays only where there was a common subset, or match the
> capabilities available to the user/operator provided constraints.
>
> deployment system. It supported different encapsulations (IPv4, IPv6, IPv4
> with IPsec, IPv6 with IPsec), but each overlay system deployed used only
> one encapsulation throughout. Users/operators indicated requirements that
> mapped directly to encapsulation configurations and we had a secure
> capability discovery system where "invitations" asked for particular
> capabilities and only nodes willing to participate responded.
>
> seems to me that this is an example that heterogeneous encaps protocols can
> be supported usefully without too much difficulty, - again, as long as each
> overlay is homogenous.
>
> Joe
>
>
>
> leading to this question for the group –Is it even feasible to proceed with
> OAM or control plane activity without assuming (or referring to) any base
> encap?
>
> Thanks
>
> Kiran
>
>
>
> *From:* nvo3 [mailto:[email-masked] <[email-masked]>] *On
> Behalf Of *Joe Touch
> *To:* Anoop Ghanwani <[email-masked]> <[email-masked]>
> *Cc:* Sam Aldrin <[email-masked]> <[email-masked]>; Bocci,
> Matthew (Nokia - GB) <[email-masked]> <[email-masked]>;
> NVO3 <[email-masked]> <[email-masked]>; Lucy yong <[email-masked]>
> <[email-masked]>; Alia Atlas <[email-masked]> <[email-masked]>
> these to sets of capabilities or fail. Using the common subset of the two
> means turning off some of the features, which avoids translation failures
> only if it can be negotiated (and both sides agree).
>
>
>

---

From: Joe Touch <[email-masked]>
Date: Mon, 10 Oct 2016 07:55:18 -0700

On 10/8/2016 10:43 AM, Greg Mirsky wrote:
> Hi Joe,
> I think that he task of OAM interworking between two NVO3 domains that
> use different dataplane encapsulations would be much simpler if their

As I noted, there are two approaches:
1) dovetail the two domains and translate between them, which supports
the subset of the capabilities of the two domains and requires translation.
2) layer a third domain over the two domains and treat the two as
independent, relying on the common third domain to establish capabilities

This is exactly the difference between pre-Internet internetworking and
Internet internetworking. It's been 47 years since we tried the Internet
experiment; it surprises me we haven't concluded whether it worked or
not yet, but if this WG wants to try the pre-Internet approach one more
time to see if it works, so be it.

Joe

---

From: Lucy yong <[email-masked]>
Date: Mon, 10 Oct 2016 18:29:47 +0000

Joe,

Thank you for the great summarization on the technical approaches.

NVO3 applications require both approaches. An NOV3 application may interwork with a network domain that is not implemented by NVO3 technology. For example, DC application in a DC provider site provides a service for an enterprise company who are on an physical infrastructure. The first approach is required to be used here. 

The second approach is useful for many applications. IMO: NVO3 WG has not worked on it from the architecture perspective. The third domain (your word, or overlay network) architecture should have edge node and transit node components and aim to use tunnels as transport link between any pair of nodes in an overlay network (third domain). Current NVO3 arch is for an NVO3 network with edge nodes only, i.e. single hop from NVO3 network perspective. 

Thanks,
Lucy
	
-----Original Message-----
From: Joe Touch [mailto:[email-masked]] 
Sent: Monday, October 10, 2016 9:55 AM
To: Greg Mirsky
Cc: Kiran.Makhijani; Anoop Ghanwani; Sam Aldrin; Bocci, Matthew (Nokia - GB); NVO3; Lucy yong; Alia Atlas
Subject: Re: [nvo3] Discussion on encapsulation formats and next steps

As I noted, there are two approaches:
1) dovetail the two domains and translate between them, which supports the subset of the capabilities of the two domains and requires translation.
2) layer a third domain over the two domains and treat the two as independent, relying on the common third domain to establish capabilities

This is exactly the difference between pre-Internet internetworking and Internet internetworking. It's been 47 years since we tried the Internet experiment; it surprises me we haven't concluded whether it worked or not yet, but if this WG wants to try the pre-Internet approach one more time to see if it works, so be it.

Joe