Subject: [nvo3] Update on encapsulation design team

From: "Bocci, Matthew (Nokia - GB)" <[email-masked]>
Date: Thu, 20 Oct 2016 08:37:06 +0000

WG,

We would like to give you an update on the process in the WG for progressing the issue of a data plane encapsulation. The chairs and Alia believe that the best way forward is to progress a single encapsulation format that addresses the technical concerns raised on the list in the recent discussions. This would address the clear overall consensus of the Berlin meeting and list for a single encapsulation.

The strategy should be to take one of the three existing encapsulations and enhance it to address these concerns. This would become the standards track output of the WG. The existing three drafts (GENEVE, GUE and VXLAN-GPE) should be forwarded to the IESG as informational after the standards track draft specifying the single encapsulation. This provides an opportunity for those encapsulations to be documented and maintained.

The single encapsulation should be viewed as one that the WG and industry can converge around for the future.

We have created a design team to progress work on a single encapsulation that can form the basis or work going forward. The design team members are: Michael Schmidt, Uri Elzur, Ilango Ganga, Erik Nordmark, Rajeev Manur, Prankaj Garg. Many thanks to these individuals for their help.

Please see below for a draft charter for the design team. Please review the charter and send comments to the list by 2nd November 2016.

Regards,

Matthew and Sam

====
NVO3 Encapsulation Design team 2016

Problem Statement
The NVO3 WG charter states that it may produce requirements for network virtualization data planes based on encapsulation of virtual network traffic over an IP-based underlay data plane. Such requirements should consider OAM and security. Based on these requirements the WG will select, extend, and/or develop one or more data plane encapsulation format(s).

This has led to drafts describing three encapsulations being adopted by the working group:
- draft-ietf-nvo3-geneve-03
- draft-ietf-nvo3-gue-04
- draft-ietf-nvo3-vxlan-gpe-02

Discussion on the list and in face-to-face meetings has identified a number of technical problems with each of these encapsulations. Furthermore, there was clear consensus at the IETF meeting in Berlin that it is undesirable for the working group to progress more than one data plane encapsulation. Although consensus could not be reached on the list, the overall consensus was for a single encapsulation (RFC2418, Section 3.3). Nonetheless there has been resistance to converging on a single encapsulation format, although doing so would provide the best benefit to the industry.

Design Team Goals
The design team should take one of the proposed encapsulations and enhance it to address the technical concerns. Backwards compatibility with the chosen encapsulation and the simple evolution of deployed networks as well as applicability to all locations in the NVO3 architecture are goals. The DT should specifically avoid a design that is burdensome on hardware implementations, but should allow future extensibility. The chosen design should also operate well with ICMP and in ECMP environments. If further extensibility is required, then it should be done in such a manner that it does not require the consent of an entity outside of the IETF.

Timeline
The design team should produce a first draft describing the proposal by end of January 2017. Target adoption by the WG by March 2017 IETF.

---

From: Fabio Maino <[email-masked]>
Date: Thu, 20 Oct 2016 12:02:50 -0700

(for full disclosure I'm one of the authors of VXLAN-GPE)

Matt, Sam, Alia,
I've expressed multiple times and in multiple venues my adversity (and 
the motivations) to set this group to design yet another encapsulation. 
I won't repeat it here once again, but I want to re-assert that it's 
still were I stand.

I've seen quite a few people in the mailing list here expressing similar 
concerns, but I see that it has not changed the opinion of the chairs 
and the AD on what they believe is the best way to move forward.

That said, here are my comments to the charter.

I think the design team first goal should be to clearly articulate the 
shortcomings of the current encapsulations proposed to the WG. This 
should be the very first deliverable of the design team. The actual 
design work should start only once the WG has reached consensus on that 
document. Especially considering that some of the encapsulations 
proposed are being deployed, I think articulating the shortcomings will 
help to make the best choice in term of (1) selecting which one will 
need to be extended, and (2) designing the actual extensions.

Below are my proposals on how to modify the wording of the charter.

On 10/20/16 1:37 AM, Bocci, Matthew (Nokia - GB) wrote:
>
> WG,
>
> We would like to give you an update on the process in the WG for 
> progressing the issue of a data plane encapsulation. The chairs and 
> Alia believe that the best way forward is to progress a single 
> encapsulation format that addresses the technical concerns raised on 
> the list in the recent discussions. This would address the clear 
> overall consensus of the Berlin meeting and list for a single 
> encapsulation.
>
> The strategy should be to take one of the three existing 
> encapsulations and enhance it to address these concerns. This would 
> become the standards track output of the WG. The existing three drafts 
> (GENEVE, GUE and VXLAN-GPE) should be forwarded to the IESG as 
> informational after the standards track draft specifying the single 
> encapsulation. This provides an opportunity for those encapsulations 
> to be documented and maintained.
>
> The single encapsulation should be viewed as one that the WG and 
> industry can converge around for the future.
>
> We have created a design team to progress work on a single 
> encapsulation that can form the basis or work going forward. The 
> design team members are: Michael Schmidt, Uri Elzur, Ilango Ganga, 
> Erik Nordmark, Rajeev Manur, Prankaj Garg. Many thanks to these 
> individuals for their help.
>
> Please see below for a draft charter for the design team. Please 
> review the charter and send comments to the list by 2^nd November 2016.
>
> Regards,
>
> Matthew and Sam
>
> ====
>
> The NVO3 WG charter states that it may produce requirements for 
> network virtualization data planes based on encapsulation of virtual 
> network traffic over an IP-based underlay data plane. Such 
> requirements should consider OAM and security. Based on these 
> requirements the WG will select, extend, and/or develop one or more 
> data plane encapsulation format(s).
>
> Discussion on the list and in face-to-face meetings has identified a 
> number of technical problems with each of these encapsulations. 
> Furthermore, there was clear consensus at the IETF meeting in Berlin 
> that it is undesirable for the working group to progress more than one 
> data plane encapsulation. Although consensus could not be reached on 
> the list, the overall consensus was for a single encapsulation 
> (RFC2418, Section 3.3). Nonetheless there has been resistance to 
> converging on a single encapsulation format, although doing so would 
> provide the best benefit to the industry.
>

The portion of the last sentence that follows the comma ("although doing 
so would provide the best benefit to the industry") doesn't seem to be 
adding anything to the charter. I'd suggest it could be removed.

> Design Team Goals
The design team should clearly articulate in a draft which are the 
shortcomings of the proposed encapsulations, and where they fall short 
in addressing the NVO3 architectural requirements.

Once the 'shortcomings' draft has reached consensus of the WG,
>
> The design team should take one of the proposed encapsulations and 
> enhance it to address the technical concerns.
>
> Backwards compatibility with the chosen encapsulation and the simple 
> evolution of deployed networks as well as applicability to all 
> locations in the NVO3 architecture
>
, together with the design goals articulated in the 'shortcoming' draft,

> are goals. The DT should specifically avoid a design that is 
> burdensome on hardware implementations, but should allow future 
> extensibility. The chosen design should also operate well with ICMP 
> and in ECMP environments. If further extensibility is required, then 
> it should be done in such a manner that it does not require the 
> consent of an entity outside of the IETF.
>
> Timeline
>
> The design team should
>
first produce the 'shortcomings' draft, get it adopted by the WG, and then

> produce a first draft describing the proposal by end of January 2017. 
> Target adoption by the WG by March 2017 IETF.
>
(those two dates may need to be adjusted accordingly)

Thanks,
Fabio

>
>
> _______________________________________________
> nvo3 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nvo3

---

From: Dino Farinacci <[email-masked]>
Date: Thu, 20 Oct 2016 12:10:29 -0700

I agree with Fabio.

Choosing a single encapsulation that is not 1 of the 3, creates a 4th one that no one wants. 

And guess what, you make all 3 authors unhappy where none of them will endorse (or implement) the 4th one.

Dino

> On Oct 20, 2016, at 12:02 PM, Fabio Maino <[email-masked]> wrote:
> 
> I've expressed multiple times and in multiple venues my adversity (and the motivations) to set this group to design yet another encapsulation. I won't repeat it here once again, but I want to re-assert that it's still were I stand. 
> 
> I've seen quite a few people in the mailing list here expressing similar concerns, but I see that it has not changed the opinion of the chairs and the AD on what they believe is the best way to move forward. 
> 
> I think the design team first goal should be to clearly articulate the shortcomings of the current encapsulations proposed to the WG. This should be the very first deliverable of the design team. The actual design work should start only once the WG has reached consensus on that document. Especially considering that some of the encapsulations proposed are being deployed, I think articulating the shortcomings will help to make the best choice in term of (1) selecting which one will need to be extended, and (2) designing the actual extensions.  
> 
> The portion of the last sentence that follows the comma ("although doing so would provide the best benefit to the industry") doesn't seem to be adding anything to the charter. I'd suggest it could be removed.  
> 
>>  
>> Design Team Goals
> The design team should clearly articulate in a draft which are the shortcomings of the proposed encapsulations, and where they fall short in addressing the NVO3 architectural requirements. 
> 
>> The design team should take one of the proposed encapsulations and enhance it to address the technical concerns. 
>> Backwards compatibility with the chosen encapsulation and the simple evolution of deployed networks as well as applicability to all locations in the NVO3 architecture
>> are goals. The DT should specifically avoid a design that is burdensome on hardware implementations, but should allow future extensibility. The chosen design should also operate well with ICMP and in ECMP environments. If further extensibility is required, then it should be done in such a manner that it does not require the consent of an entity outside of the IETF.
>>  
>> Timeline
>> The design team should 
>> produce a first draft describing the proposal by end of January 2017. Target adoption by the WG by March 2017 IETF.
>>

---

From: Tom Herbert <[email-masked]>
Date: Thu, 20 Oct 2016 12:35:40 -0700

> And guess what, you make all 3 authors unhappy where none of them will endorse (or implement) the 4th one.
My $0.02: That's not the way I read Matthew's message. It seems like
the conclusion to the technical objections query is that objections
were raised for all three protocols and so none of them were ready for
standardization. The goal of the design team seems to be to start with
one, presumably the one with the fewest issues, and enhance it to
answer all the technical objections with an effort to maintain
backwards compatibility for that protocol. This might essentially be a
method of picking one as I believe you proposed earlier.

Tom

---

From: Dino Farinacci <[email-masked]>
Date: Thu, 20 Oct 2016 12:54:14 -0700

Tom, we'll see what the design team comes up with.

Dino

> On Oct 20, 2016, at 12:35 PM, Tom Herbert <[email-masked]> wrote:
>

---

From: Anoop Ghanwani <[email-masked]>
Date: Mon, 24 Oct 2016 11:26:07 -0700

Agree with Fabio (including the suggestion for an interim deliverable on
shortcomings).  If the WG doesn't agree on the shortcomings, chances are
they may not like the 4th encap.

Anoop

On Thu, Oct 20, 2016 at 12:10 PM, Dino Farinacci <[email-masked]>
wrote:

> the motivations) to set this group to design yet another encapsulation. I
> won't repeat it here once again, but I want to re-assert that it's still
> were I stand.
> >
> concerns, but I see that it has not changed the opinion of the chairs and
> the AD on what they believe is the best way to move forward.
> >
> shortcomings of the current encapsulations proposed to the WG. This should
> be the very first deliverable of the design team. The actual design work
> should start only once the WG has reached consensus on that document.
> Especially considering that some of the encapsulations proposed are being
> deployed, I think articulating the shortcomings will help to make the best
> choice in term of (1) selecting which one will need to be extended, and (2)
> designing the actual extensions.
> >
> progressing the issue of a data plane encapsulation. The chairs and Alia
> believe that the best way forward is to progress a single encapsulation
> format that addresses the technical concerns raised on the list in the
> recent discussions. This would address the clear overall consensus of the
> Berlin meeting and list for a single encapsulation.
> >>
> >> The strategy should be to take one of the three existing encapsulations
> and enhance it to address these concerns. This would become the standards
> track output of the WG. The existing three drafts (GENEVE, GUE and
> VXLAN-GPE) should be forwarded to the IESG as informational after the
> standards track draft specifying the single encapsulation. This provides an
> opportunity for those encapsulations to be documented and maintained.
> >>
> encapsulation that can form the basis or work going forward. The design
> team members are: Michael Schmidt, Uri Elzur, Ilango Ganga, Erik Nordmark,
> Rajeev Manur, Prankaj Garg. Many thanks to these individuals for their help.
> >>
> >> Please see below for a draft charter for the design team. Please review
> the charter and send comments to the list by 2nd November 2016.
> >>
> >> Regards,
> >>
> >> Matthew and Sam
> >>
> >>
> >> ====
> >> The NVO3 WG charter states that it may produce requirements for network
> virtualization data planes based on encapsulation of virtual network
> traffic over an IP-based underlay data plane. Such requirements should
> consider OAM and security. Based on these requirements the WG will select,
> extend, and/or develop one or more data plane encapsulation format(s).
> >>
> Furthermore, there was clear consensus at the IETF meeting in Berlin that
> it is undesirable for the working group to progress more than one data
> plane encapsulation. Although consensus could not be reached on the list,
> the overall consensus was for a single encapsulation (RFC2418, Section
> 3.3). Nonetheless there has been resistance to converging on a single
> encapsulation format, although doing so would provide the best benefit to
> the industry.
> >
> shortcomings of the proposed encapsulations, and where they fall short in
> addressing the NVO3 architectural requirements.
> >
> evolution of deployed networks as well as applicability to all locations in
> the NVO3 architecture
> >> are goals. The DT should specifically avoid a design that is burdensome
> on hardware implementations, but should allow future extensibility. The
> chosen design should also operate well with ICMP and in ECMP environments.
> If further extensibility is required, then it should be done in such a
> manner that it does not require the consent of an entity outside of the
> IETF.
> >>
> >> Timeline
> >> The design team should

---

From: Alia Atlas <[email-masked]>
Date: Mon, 24 Oct 2016 14:30:05 -0400

Anoop & Fabiio,

> they may not like the 4th encap.

What do you expect to be different from the summary of technical objections
that came out of the last discussion?  Are you looking for more detail?

I didn't see disagreement about the accuracy of the technical objections.

Regards,
Alia

>> > I've seen quite a few people in the mailing list here expressing
>> similar concerns, but I see that it has not changed the opinion of the
>> chairs and the AD on what they believe is the best way to move forward.
>> >
>> encapsulations and enhance it to address these concerns. This would become
>> the standards track output of the WG. The existing three drafts (GENEVE,
>> GUE and VXLAN-GPE) should be forwarded to the IESG as informational after
>> the standards track draft specifying the single encapsulation. This
>> provides an opportunity for those encapsulations to be documented and
>> maintained.
>> >>
>> review the charter and send comments to the list by 2nd November 2016.
>> >>
>> >> Regards,
>> >>
>> >> Matthew and Sam
>> >>
>> >>
>> >> ====
>> network traffic over an IP-based underlay data plane. Such requirements
>> should consider OAM and security. Based on these requirements the WG will
>> select, extend, and/or develop one or more data plane encapsulation
>> format(s).
>> >>
>> > The portion of the last sentence that follows the comma ("although
>> doing so would provide the best benefit to the industry") doesn't seem to
>> be adding anything to the charter. I'd suggest it could be removed.
>> >
>> >>
>> >> Design Team Goals
>> extensibility. The chosen design should also operate well with ICMP and in
>> ECMP environments. If further extensibility is required, then it should be
>> done in such a manner that it does not require the consent of an entity
>> outside of the IETF.
>> >>
>> >> Timeline
>> >> The design team should

---

From: Anoop Ghanwani <[email-masked]>
Date: Mon, 24 Oct 2016 11:43:50 -0700

Alia,

I think it will provide an official reference which will be helpful since
all the encaps will be around a long time and we can point people to that
document when we are asked the question "why did they develop yet another
encap?".

Thanks,
Anoop

On Mon, Oct 24, 2016 at 11:30 AM, Alia Atlas <[email-masked]> wrote:

> What do you expect to be different from the summary of technical
> objections that came out of the last discussion?  Are you looking for more
> detail?
>

---

From: Alia Atlas <[email-masked]>
Date: Mon, 24 Oct 2016 14:46:15 -0400

Hi Anoop,

> document when we are asked the question "why did they develop yet another

So what you are asking for is a section in the overall draft that talks
about the motivations and improvements?
There's a trade-off of speed and getting a solution done versus doing
process work for a theoretical future that won't happen if we don't get the
technical work finished.

Regards,
Alia

>>>> > I've expressed multiple times and in multiple venues my adversity
>>>> (and the motivations) to set this group to design yet another
>>>> encapsulation. I won't repeat it here once again, but I want to re-assert
>>>> that it's still were I stand.
>>>> >
>>>> > I think the design team first goal should be to clearly articulate
>>>> the shortcomings of the current encapsulations proposed to the WG. This
>>>> should be the very first deliverable of the design team. The actual design
>>>> work should start only once the WG has reached consensus on that document.
>>>> >> produce a first draft describing the proposal by end of January
>>>> 2017. Target adoption by the WG by March 2017 IETF.
>>>> >>

---

From: Anoop Ghanwani <[email-masked]>
Date: Mon, 24 Oct 2016 12:51:45 -0700

Hi Alia,

Ideally I would rather not go there at all as expressed in the first part
of Fabio's note.  Assuming we must go there, having a section that
documents why the WG thought we needed a new one would be good enough for
me.

Anoop

On Mon, Oct 24, 2016 at 11:46 AM, Alia Atlas <[email-masked]> wrote:

>>>> Agree with Fabio (including the suggestion for an interim deliverable
>>>> on shortcomings).  If the WG doesn't agree on the shortcomings, chances are
>>>>> >> Discussion on the list and in face-to-face meetings has identified
>>>>> a number of technical problems with each of these encapsulations.
>>>>> >> Backwards compatibility with the chosen encapsulation and the
>>>>> simple evolution of deployed networks as well as applicability to all

---

From: Fabio Maino <[email-masked]>
Date: Mon, 24 Oct 2016 12:54:35 -0700

I think it will be useful for the design team to articulate clearly and 
motivate those objections, and present it to the WG for formal 
discussion BEFORE starting the new design. That should help consolidate 
the requirements for the design of the new encap.

A separate technical document will help  evaluate in depth the 
objections raised, rather than rely on hundreds of emails and their 
different interpretation.

I think I have seen quite a lot of disagreement around the objection 
themselves.

For example, wrt GPE extensibility the summary reported that

"- GPE is insufficiently extensible. Numerous extensions and options 
have been designed for GUE and Geneve. Note that these have not yet been 
validated by the WG."

That assertion seems to ignore (as it has been pointed out multiple time 
in the mailing list) what has been done with NSH, or other proposed  
encodings for carrying metadata with VXLAN-GPE such as:
https://tools.ietf.org/html/draft-ietf-sfc-nsh-10
https://tools.ietf.org/html/draft-brockners-inband-oam-data

I would like the WG to articulate why, from a technical point of view, 
proper layering can't be used to extend GPE (as recommended by the GPE 
draft itself), as it is certainly possible (and done) in the two drafts 
indicated above.

Same goes for the security objection to GPE.

I'm quite sure the authors of the other two proposals have similar view 
on the objections raised to their proposals.

A separate document, in my opinion, would help understanding what needs 
to be addressed before we start with the new design.

Thanks,
Fabio

On 10/24/16 12:51 PM, Anoop Ghanwani wrote:
> Hi Alia,
>
> Ideally I would rather not go there at all as expressed in the first 
> part of Fabio's note.  Assuming we must go there, having a section 
> that documents why the WG thought we needed a new one would be good 
> enough for me.
>
> Anoop
>
> On Mon, Oct 24, 2016 at 11:46 AM, Alia Atlas <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>     Hi Anoop,
>
>     On Mon, Oct 24, 2016 at 2:43 PM, Anoop Ghanwani
>     <[email-masked] <mailto:[email-masked]>> wrote:
>
>         Alia,
>
>         I think it will provide an official reference which will be
>         helpful since all the encaps will be around a long time and we
>         can point people to that document when we are asked the
>         question "why did they develop yet another encap?".
>
>
>     So what you are asking for is a section in the overall draft that
>     talks about the motivations and improvements?
>     There's a trade-off of speed and getting a solution done versus
>     doing process work for a theoretical future that won't happen if
>     we don't get the technical work finished.
>
>     Regards,
>     Alia
>
>         Thanks,
>         Anoop
>
>         On Mon, Oct 24, 2016 at 11:30 AM, Alia Atlas
>         <[email-masked] <mailto:[email-masked]>> wrote:
>
>             Anoop & Fabiio,
>
>             On Mon, Oct 24, 2016 at 2:26 PM, Anoop Ghanwani
>             <[email-masked] <mailto:[email-masked]>> wrote:
>
>                 Agree with Fabio (including the suggestion for an
>                 interim deliverable on shortcomings).  If the WG
>                 doesn't agree on the shortcomings, chances are they
>                 may not like the 4th encap.
>
>
>             What do you expect to be different from the summary of
>             technical objections that came out of the last
>             discussion?  Are you looking for more detail?
>                 On Thu, Oct 20, 2016 at 12:10 PM, Dino Farinacci
>                 <[email-masked] <mailto:[email-masked]>> wrote:
>
>                     I agree with Fabio.
>
>                     Choosing a single encapsulation that is not 1 of
>                     the 3, creates a 4th one that no one wants.
>
>                     And guess what, you make all 3 authors unhappy
>                     where none of them will endorse (or implement) the
>                     4th one.
>
>                     Dino
>
>                     > On Oct 20, 2016, at 12:02 PM, Fabio Maino
>                     <[email-masked] <mailto:[email-masked]>> wrote:
>                     >
>                     > I've expressed multiple times and in multiple
>                     venues my adversity (and the motivations) to set
>                     this group to design yet another encapsulation. I
>                     won't repeat it here once again, but I want to
>                     re-assert that it's still were I stand.
>                     >
>                     > I've seen quite a few people in the mailing list
>                     here expressing similar concerns, but I see that
>                     it has not changed the opinion of the chairs and
>                     > I think the design team first goal should be to
>                     clearly articulate the shortcomings of the current
>                     encapsulations proposed to the WG. This should be
>                     the very first deliverable of the design team. The
>                     actual design work should start only once the WG
>                     has reached consensus on that document. Especially
>                     considering that some of the encapsulations
>                     proposed are being deployed, I think articulating
>                     the shortcomings will help to make the best choice
>                     in term of (1) selecting which one will need to be
>                     extended, and (2) designing the actual extensions.
>                     >
>                     >> We would like to give you an update on the
>                     process in the WG for progressing the issue of a
>                     data plane encapsulation. The chairs and Alia
>                     believe that the best way forward is to progress a
>                     single encapsulation format that addresses the
>                     technical concerns raised on the list in the
>                     recent discussions. This would address the clear
>                     overall consensus of the Berlin meeting and list
>                     for a single encapsulation.
>                     >>
>                     >> The strategy should be to take one of the three
>                     existing encapsulations and enhance it to address
>                     these concerns. This would become the standards
>                     track output of the WG. The existing three drafts
>                     (GENEVE, GUE and VXLAN-GPE) should be forwarded to
>                     the IESG as informational after the standards
>                     track draft specifying the single encapsulation.
>                     This provides an opportunity for those
>                     encapsulations to be documented and maintained.
>                     >>
>                     >> The single encapsulation should be viewed as
>                     one that the WG and industry can converge around
>                     for the future.
>                     >>
>                     >> We have created a design team to progress work
>                     on a single encapsulation that can form the basis
>                     or work going forward. The design team members
>                     are: Michael Schmidt, Uri Elzur, Ilango Ganga,
>                     Erik Nordmark, Rajeev Manur, Prankaj Garg. Many
>                     thanks to these individuals for their help.
>                     >>
>                     >> Please see below for a draft charter for the
>                     design team. Please review the charter and send
>                     comments to the list by 2nd November 2016.
>                     >>
>                     >> Regards,
>                     >>
>                     >> Matthew and Sam
>                     >>
>                     >>
>                     >> ====
>                     >> The NVO3 WG charter states that it may produce
>                     requirements for network virtualization data
>                     planes based on encapsulation of virtual network
>                     traffic over an IP-based underlay data plane. Such
>                     requirements should consider OAM and security.
>                     Based on these requirements the WG will select,
>                     >> This has led to drafts describing three
>                     encapsulations being adopted by the working group:
>                     >> Discussion on the list and in face-to-face
>                     meetings has identified a number of technical
>                     problems with each of these encapsulations.
>                     Furthermore, there was clear consensus at the IETF
>                     meeting in Berlin that it is undesirable for the
>                     working group to progress more than one data plane
>                     encapsulation. Although consensus could not be
>                     reached on the list, the overall consensus was for
>                     a single encapsulation (RFC2418, Section 3.3).
>                     Nonetheless there has been resistance to
>                     converging on a single encapsulation format,
>                     although doing so would provide the best benefit
>                     to the industry.
>                     >
>                     > The portion of the last sentence that follows
>                     the comma ("although doing so would provide the
>                     best benefit to the industry") doesn't seem to be
>                     > The design team should clearly articulate in a
>                     draft which are the shortcomings of the proposed
>                     encapsulations, and where they fall short in
>                     >> The design team should take one of the proposed
>                     encapsulations and enhance it to address the
>                     technical concerns.
>                     >> Backwards compatibility with the chosen
>                     encapsulation and the simple evolution of deployed
>                     networks as well as applicability to all locations
>                     in the NVO3 architecture
>                     >> are goals. The DT should specifically avoid a
>                     design that is burdensome on hardware
>                     implementations, but should allow future
>                     extensibility. The chosen design should also
>                     operate well with ICMP and in ECMP environments.
>                     If further extensibility is required, then it
>                     should be done in such a manner that it does not
>                     require the consent of an entity outside of the IETF.
>                     >>
>                     >> Timeline
>                     >> The design team should
>                     >> produce a first draft describing the proposal
>                     by end of January 2017. Target adoption by the WG
>                     by March 2017 IETF.
>                     >>
>                     >> [email-masked] <mailto:[email-masked]>
>                     <https://www.ietf.org/mailman/listinfo/nvo3>
>                     >
>                     >
>                     > [email-masked] <mailto:[email-masked]>
>                     <https://www.ietf.org/mailman/listinfo/nvo3>
>
>                     [email-masked] <mailto:[email-masked]>
>                     <https://www.ietf.org/mailman/listinfo/nvo3>
>
>
>
>                 [email-masked] <mailto:[email-masked]>
>                 <https://www.ietf.org/mailman/listinfo/nvo3>
>
>
>
>
>

---

From: Tom Herbert <[email-masked]>
Date: Mon, 24 Oct 2016 13:36:31 -0700

On Mon, Oct 24, 2016 at 12:54 PM, Fabio Maino <[email-masked]> wrote:
> motivate those objections, and present it to the WG for formal discussion
> BEFORE starting the new design. That should help consolidate the
> requirements for the design of the new encap.
>
> A separate technical document will help  evaluate in depth the objections
> raised, rather than rely on hundreds of emails and their different
> interpretation.
>
> "- GPE is insufficiently extensible. Numerous extensions and options have
> been designed for GUE and Geneve. Note that these have not yet been
> validated by the WG."
>
> That assertion seems to ignore (as it has been pointed out multiple time in
> the mailing list) what has been done with NSH, or other proposed  encodings
> for carrying metadata with VXLAN-GPE such as:
> proper layering can't be used to extend GPE (as recommended by the GPE draft
> itself), as it is certainly possible (and done) in the two drafts indicated
> above.
>
Fabio,

I disagree that the question of extensibility is adequately answered
in VXLAN-GPE by just referring to NSH. Reasons are:

1) There is no proposal or real example of what an extension for nvo3
encapsulation would actually look like. I have asked many times that
VXLAN-GPE authors provide something, for instance an example of a
security token, but to date nothing specific has been produced. I have
a similar issue with Geneve since they have not formally proposed any
extensions, but at list it is clear in Geneve what the format and
general processing of extensions would be.

2) If we accept that NSH is the answer to extensibility in VXLAN-GPE
then we need to amend the technical objections. in particular, there
were technical objections raised for Geneve and GUE that they were
more complex than VXLAN-GPE because of the extensibility mechanism. If
NSH is considered part of VLXAN-GPE then we also need to consider the
feasibility of using NSH in an NV environment and answer questions
around hardware friendliness, susceptibility to DDOS, etc. Again,
without any sample extensions or specification of how this works with
VXLAN_GPE the merits of extensibility seem really hard to evaluate.

Thanks,
Tom

> I'm quite sure the authors of the other two proposals have similar view on
> the objections raised to their proposals.
>
> A separate document, in my opinion, would help understanding what needs to
> be addressed before we start with the new design.
>
>
> Thanks,
> Fabio
>
>
> Ideally I would rather not go there at all as expressed in the first part of
> Fabio's note.  Assuming we must go there, having a section that documents
> why the WG thought we needed a new one would be good enough for me.
>
> Anoop
>
>>>>>> > (and the motivations) to set this group to design yet another encapsulation.
>>>>>> > I won't repeat it here once again, but I want to re-assert that it's still
>>>>>> > were I stand.
>>>>>> >
>>>>>> >> the standards track output of the WG. The existing three drafts (GENEVE, GUE
>>>>>> >> and VXLAN-GPE) should be forwarded to the IESG as informational after the
>>>>>> >> encapsulation that can form the basis or work going forward. The design team
>>>>>> >> members are: Michael Schmidt, Uri Elzur, Ilango Ganga, Erik Nordmark, Rajeev
>>>>>> >> Manur, Prankaj Garg. Many thanks to these individuals for their help.
>>>>>> >>
>>>>>> >> network virtualization data planes based on encapsulation of virtual network
>>>>>> >> Furthermore, there was clear consensus at the IETF meeting in Berlin that it
>>>>>> >> is undesirable for the working group to progress more than one data plane
>>>>>> >> encapsulation. Although consensus could not be reached on the list, the
>>>>>> >> overall consensus was for a single encapsulation (RFC2418, Section 3.3).
>>>>>> >> Nonetheless there has been resistance to converging on a single
>>>>>> > doing so would provide the best benefit to the industry") doesn't seem to be

---

From: Alia Atlas <[email-masked]>
Date: Mon, 24 Oct 2016 16:44:26 -0400

Fabio,

> raised, rather than rely on hundreds of emails and their different

The WG raised the various objections and I have not seen any efforts to
claim that specific ones are not accurate technical objections nor that the
WG should ignore particular objections.

> draft itself), as it is certainly possible (and done) in the two drafts

It is not the responsibility of the WG to explain why different approaches
that were not even written down in detail (i.e. an internet-draft) are not
acceptable solutions.  If you wish to claim that layering with NSH is a
solution, that may be something interesting to discuss.  However, NSH
provides for both fixed MD-Type 1 meta-data as well as MD-Type 2 meta-data;
it assumes a control-plane that communicates what meta-data is to be
carried and how.    Obviously, an approach that only supports MD-Type 1
meta-data will not have the flexibility that I believe was requested for
extensibility.

I will note that any issues around properly attributing ICMP messages to
the triggering sender will still exist with large headers.  I do not
currently understand, from your suggestion, what would be different as far
as concerns about header size and processing by having extra bytes for the
NSH header.

I think we are all quite familiar with the alleged simplicity of the
hand-waved design option.

> Same goes for the security objection to GPE.

Frankly, I can't parse this other than assuming you mean that security is
not required.  Again, if you do not think the WG should agree with this
technical objection, then you need to be able to clearly explain why and
see if there is WG consensus.

> the objections raised to their proposals.

Then they - and others - are and have been welcome to clearly express on
the NVO3 mailing list why those objections should be ignored.   That is an
option - if there is WG consensus on it.  PLEASE reread RFC 7282.

> be addressed before we start with the new design.

Do you want to have an IETF standard in this space?

I understand the desire to stall progress in favor of  proprietary
implementations.
I would prefer that the IETF not fail to provide standards in this space.
I am not willing to leave open a WG that is determinedly failing to make
progress on its charter.

Regards,
Alia

>>> I think it will provide an official reference which will be helpful
>>> since all the encaps will be around a long time and we can point people to
>>> that document when we are asked the question "why did they develop yet
>>> another encap?".
>>>
>>

---

From: Fabio Maino <[email-masked]>
Date: Mon, 24 Oct 2016 17:46:19 -0700

Hi Alia,
please see below.

On 10/24/16 1:44 PM, Alia Atlas wrote:
> Fabio,
>
> On Mon, Oct 24, 2016 at 3:54 PM, Fabio Maino <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>
>     I think it will be useful for the design team to articulate
>     clearly and motivate those objections, and present it to the WG
>     for formal discussion BEFORE starting the new design. That should
>     help consolidate the requirements for the design of the new encap.
>
>     objections raised, rather than rely on hundreds of emails and
>     their different interpretation.
>
>
> The WG raised the various objections and I have not seen any efforts 
> to claim that specific ones are not accurate technical objections nor 
> that the WG should ignore particular objections.
>
>     I think I have seen quite a lot of disagreement around the
>     objection themselves.
>
>     "- GPE is insufficiently extensible. Numerous extensions and
>     options have been designed for GUE and Geneve. Note that these
>     have not yet been validated by the WG."
>
>     That assertion seems to ignore (as it has been pointed out
>     multiple time in the mailing list) what has been done with NSH, or
>     other proposed  encodings for carrying metadata with VXLAN-GPE
>     such as:
>     <https://tools.ietf.org/html/draft-ietf-sfc-nsh-10>
>     <https://tools.ietf.org/html/draft-brockners-inband-oam-data>
>
>     I would like the WG to articulate why, from a technical point of
>     view, proper layering can't be used to extend GPE (as recommended
>     by the GPE draft itself), as it is certainly possible (and done)
>     in the two drafts indicated above.
>
>
> It is not the responsibility of the WG to explain why different 
> approaches that were not even written down in detail (i.e. an 
> internet-draft) are not acceptable solutions.  If you wish to claim 
> that layering with NSH is a solution, that may be something 
> interesting to discuss. However, NSH provides for both fixed MD-Type 1 
> meta-data as well as MD-Type 2 meta-data; it assumes a control-plane 
> that communicates what meta-data is to be carried and how.   
>  Obviously, an approach that only supports MD-Type 1 meta-data will 
> not have the flexibility that I believe was requested for extensibility.

So, an additional requirement that I infer from your text above is that 
the encap should flexibly allow transport of metadata, even in the 
absence of a control plane. I can't see that requirement spelled out in 
the summary.

This is one example of what I hope will be part of a detailed analysis 
of what are the requirements of the encap that should be designed. This 
will also help determining what requirements the proposed encaps are not 
matching.

>
> I will note that any issues around properly attributing ICMP messages 
> to the triggering sender will still exist with large headers.  I do 
> not currently understand, from your suggestion, what would be 
> different as far as concerns about header size and processing by 
> having extra bytes for the NSH header.
>

Right, that's why I'm suggesting to avoid to hand-wave requirements.

> Frankly, I can't parse this other than assuming you mean that security 
> is not required.  Again, if you do not think the WG should agree with 
> this technical objection, then you need to be able to clearly explain 
> why and see if there is WG consensus.
>
>     I'm quite sure the authors of the other two proposals have similar
>     view on the objections raised to their proposals.
>
>
> Then they - and others - are and have been welcome to clearly express 
> on the NVO3 mailing list why those objections should be ignored.   
> That is an option - if there is WG consensus on it.  PLEASE reread RFC 
> 7282.
>
>     A separate document, in my opinion, would help understanding what
>     needs to be addressed before we start with the new design.
>
>

It doesn't have to become a standard, but I think it should be done, and 
discussed in the WG, before the design begins. Having it as a separate 
draft might help. If then you want to merge it or morph it into the 
design doc that will be fine.

> I understand the desire to stall progress in favor of  proprietary 

If you are referring to my desires, I'm afraid you are grossly 
misunderstanding.

> I would prefer that the IETF not fail to provide standards in this space.

Right, we just happen to disagree on how those standards would look 
like, and how we can get there.

FWIW, I'm  just answering to the call of the chairs to "hear opinions 
and confirmation or disagreement on interest in creating a DP 
encapsulation that addresses the various technical concerns".

Thanks,
Fabio

> I am not willing to leave open a WG that is determinedly failing to 
> make progress on its charter.
>
> Regards,
> Alia
>
>
>     Thanks,
>     Fabio
>
>
>>     Ideally I would rather not go there at all as expressed in the
>>     first part of Fabio's note.  Assuming we must go there, having a
>>     section that documents why the WG thought we needed a new one
>>     would be good enough for me.
>>
>>     Anoop
>>
>>             I think it will provide an official reference which will
>>             be helpful since all the encaps will be around a long
>>             time and we can point people to that document when we are
>>             asked the question "why did they develop yet another encap?".
>>
>>
>>         So what you are asking for is a section in the overall draft
>>         that talks about the motivations and improvements?
>>         There's a trade-off of speed and getting a solution done
>>         versus doing process work for a theoretical future that won't
>>         happen if we don't get the technical work finished.
>>
>>         Regards,
>>         Alia
>>
>>             Thanks,
>>             Anoop
>>
>>                     doesn't agree on the shortcomings, chances are
>>                 What do you expect to be different from the summary
>>                 of technical objections that came out of the last
>>                         Choosing a single encapsulation that is not 1
>>                         of the 3, creates a 4th one that no one wants.
>>
>>                         And guess what, you make all 3 authors
>>                         unhappy where none of them will endorse (or
>>                         implement) the 4th one.
>>
>>                         Dino
>>
>>                         > I've expressed multiple times and in
>>                         multiple venues my adversity (and the
>>                         motivations) to set this group to design yet
>>                         another encapsulation. I won't repeat it here
>>                         once again, but I want to re-assert that it's
>>                         still were I stand.
>>                         >
>>                         > I've seen quite a few people in the mailing
>>                         list here expressing similar concerns, but I
>>                         see that it has not changed the opinion of
>>                         the chairs and the AD on what they believe is
>>                         the best way to move forward.
>>                         >
>>                         > I think the design team first goal should
>>                         be to clearly articulate the shortcomings of
>>                         the current encapsulations proposed to the
>>                         WG. This should be the very first deliverable
>>                         of the design team. The actual design work
>>                         should start only once the WG has reached
>>                         consensus on that document. Especially
>>                         proposed are being deployed, I think
>>                         articulating the shortcomings will help to
>>                         make the best choice in term of (1) selecting
>>                         which one will need to be extended, and (2)
>>                         process in the WG for progressing the issue
>>                         of a data plane encapsulation. The chairs and
>>                         Alia believe that the best way forward is to
>>                         progress a single encapsulation format that
>>                         addresses the technical concerns raised on
>>                         the list in the recent discussions. This
>>                         would address the clear overall consensus of
>>                         the Berlin meeting and list for a single
>>                         encapsulation.
>>                         >>
>>                         >> The strategy should be to take one of the
>>                         three existing encapsulations and enhance it
>>                         to address these concerns. This would become
>>                         the standards track output of the WG. The
>>                         existing three drafts (GENEVE, GUE and
>>                         VXLAN-GPE) should be forwarded to the IESG as
>>                         informational after the standards track draft
>>                         specifying the single encapsulation. This
>>                         provides an opportunity for those
>>                         >> The single encapsulation should be viewed
>>                         as one that the WG and industry can converge
>>                         around for the future.
>>                         >>
>>                         >> We have created a design team to progress
>>                         work on a single encapsulation that can form
>>                         the basis or work going forward. The design
>>                         team members are: Michael Schmidt, Uri Elzur,
>>                         Ilango Ganga, Erik Nordmark, Rajeev Manur,
>>                         Prankaj Garg. Many thanks to these
>>                         >> Please see below for a draft charter for
>>                         the design team. Please review the charter
>>                         and send comments to the list by 2nd November
>>                         2016.
>>                         >>
>>                         >> Regards,
>>                         >>
>>                         >> Matthew and Sam
>>                         >>
>>                         >>
>>                         >> ====
>>                         virtualization data planes based on
>>                         encapsulation of virtual network traffic over
>>                         an IP-based underlay data plane. Such
>>                         requirements should consider OAM and
>>                         security. Based on these requirements the WG
>>                         will select, extend, and/or develop one or
>>                         more data plane encapsulation format(s).
>>                         >>
>>                         Furthermore, there was clear consensus at the
>>                         IETF meeting in Berlin that it is undesirable
>>                         for the working group to progress more than
>>                         one data plane encapsulation. Although
>>                         consensus could not be reached on the list,
>>                         the overall consensus was for a single
>>                         encapsulation (RFC2418, Section 3.3).
>>                         although doing so would provide the best
>>                         benefit to the industry.
>>                         >
>>                         > The portion of the last sentence that
>>                         follows the comma ("although doing so would
>>                         provide the best benefit to the industry")
>>                         doesn't seem to be adding anything to the
>>                         charter. I'd suggest it could be removed.
>>                         >
>>                         >>
>>                         >> Design Team Goals
>>                         > The design team should clearly articulate
>>                         in a draft which are the shortcomings of the
>>                         proposed encapsulations, and where they fall
>>                         short in addressing the NVO3 architectural
>>                         requirements.
>>                         >
>>                         >> The design team should take one of the
>>                         proposed encapsulations and enhance it to
>>                         address the technical concerns.
>>                         encapsulation and the simple evolution of
>>                         deployed networks as well as applicability to
>>                         all locations in the NVO3 architecture
>>                         >> are goals. The DT should specifically
>>                         avoid a design that is burdensome on hardware
>>                         operate well with ICMP and in ECMP
>>                         environments. If further extensibility is
>>                         required, then it should be done in such a
>>                         manner that it does not require the consent
>>                         of an entity outside of the IETF.
>>                         >>
>>                         >> Timeline
>>                         >> The design team should
>>                         >> produce a first draft describing the
>>                         proposal by end of January 2017. Target
>>                         adoption by the WG by March 2017 IETF.
>>                         >>

---

From: Alia Atlas <[email-masked]>
Date: Mon, 24 Oct 2016 21:31:30 -0400

Hi Fabio,

On Mon, Oct 24, 2016 at 8:46 PM, Fabio Maino <[email-masked]> wrote:

> the encap should flexibly allow transport of metadata, even in the absence
> of a control plane. I can't see that requirement spelled out in the
> summary.
>

I think that the encap needs to handle the inter-data-center cases as well
as the intra-data-center cases.  I would have preferred more detail in the
technical objections myself - and I'm certain that the design team is
capable of coming back with questions.

> This is one example of what I hope will be part of a detailed analysis of
> what are the requirements of the encap that should be designed. This will
> also help determining what requirements the proposed encaps are not
> matching.
>

This WG has already tried to do a gap analysis and determine requirements.
That work did not go well or swiftly.  If there are already expressed
requirements - and I know that you have concerns about what "arbitrary
extensibility" really needs to handle - having discussions there is useful.

I don't think that it will help the WG or design team to reopen all
requirements or expand the set.  This DT is to resolve the WG's lack of
consensus on an acceptable encap.

> Right, that's why I'm suggesting to avoid to hand-wave requirements.

 But you were asking for the WG to explain why a hand-waved idea (VXLAN-GPE
on top of NSH) isn't the right answer.

> draft might help. If then you want to merge it or morph it into the design
> doc that will be fine.
>

When I was asking about a standard in this space, I mean a standard
encapsulation for the data-center virtualization.  I do understand that
there are those who have participated and do not think a standard is a good
idea.   What wasn't clear to me is where you are on that spectrum.  It is
also challenging to distinguish attitudes like "good idea but I don't see
how we'll ever get there".

> If you are referring to my desires, I'm afraid you are grossly

I am pleased to hear that.  When I see suggestions for adding significant
process work before the WG can agree to something that can be implemented,
it doesn't seem promising to me.  I have, obviously, talked to a number of
people about the NVO3 encapsulations and part of what makes this process
challenging is the groups dedicated to certain encaps that are already
implemented.

> Right, we just happen to disagree on how those standards would look like,
> and how we can get there.
>

I am always quite happy to listen to your or others ideas about how we can
get there.  We are all working together to do so - and I'd be delighted to
get some more ideas on how to help NVO3 succeed.

I don't see how we disagree on what the standards would look like, since I
don't actually have or intend to express a particular technical opinion.  I
do sometimes see issues that will cause problems in the architecture.

> FWIW, I'm  just answering to the call of the chairs to "hear opinions and
> confirmation or disagreement on interest in creating a DP encapsulation
> that addresses the various technical concerns".
>

Absolutely!  Your contribution is welcome.

Regards,
Alia

Thanks,
>>>>> On Mon, Oct 24, 2016 at 2:26 PM, Anoop Ghanwani <[email-masked]
>>>>> > wrote:
>>>>>
>>>>>> On Thu, Oct 20, 2016 at 12:10 PM, Dino Farinacci <[email-masked]
>>>>>> > wrote:
>>>>>>
>>>>>>> I agree with Fabio.
>>>>>>>
>>>>>>> >> Discussion on the list and in face-to-face meetings has
>>>>>>> identified a number of technical problems with each of these
>>>>>>> encapsulations. Furthermore, there was clear consensus at the IETF meeting
>>>>>>> in Berlin that it is undesirable for the working group to progress more
>>>>>>> than one data plane encapsulation. Although consensus could not be reached
>>>>>>> on the list, the overall consensus was for a single encapsulation (RFC2418,
>>>>>>> Section 3.3). Nonetheless there has been resistance to converging on a
>>>>>>> single encapsulation format, although doing so would provide the best
>>>>>>> benefit to the industry.
>>>>>>> >
>>>>>>> >> The design team should take one of the proposed encapsulations
>>>>>>> and enhance it to address the technical concerns.

---

From: Tom Herbert <[email-masked]>
Date: Tue, 25 Oct 2016 09:02:13 -0700

On Mon, Oct 24, 2016 at 5:46 PM, Fabio Maino <[email-masked]> wrote:
> it assumes a control-plane that communicates what meta-data is to be carried
> and how.    Obviously, an approach that only supports MD-Type 1 meta-data
> will not have the flexibility that I believe was requested for
> extensibility.
>
>
> So, an additional requirement that I infer from your text above is that the
> encap should flexibly allow transport of metadata, even in the absence of a
> control plane. I can't see that requirement spelled out in the summary.
>
> I will note that any issues around properly attributing ICMP messages to the
> triggering sender will still exist with large headers.  I do not currently
> understand, from your suggestion, what would be different as far as concerns
> about header size and processing by having extra bytes for the NSH header.
>
>
> technical objection, then you need to be able to clearly explain why and see
> if there is WG consensus.
>
The need for extensibility came out of the requirements for nvo3
security which are listed in draft-ietf-nvo3-security-requirements.
We already had a discussion on the security objections raised for
VXLAN-GPE in the "Security in VXLAN-GPE" thread (security concerns
about VXLAN have also been raised elsewhere also
http://blog.ipspace.net/2015/04/omg-vxlan-encapsulation-has-no-security.html).
I still don't see how data plane security requirements can be
satisfied in VLXAN-GPE-- again the "just use NSH" argument provides
nothing substantive for a protocol specification. Neither do I believe
this is something we can plan to address down the road, security is a
real concern now for those of us deploying protocols at scale.

I was assuming that the design team will carefully consider how to
best meet the requirements around security and the closely related
issues around DOS attacks (hopefully last week's attacks are a wakeup
call to this), but maybe this should be explicit in the DT charter to
avoid confusion. IMO getting security right is far more important than
the goal of trying to maintain compatibility with existing protocols.

Tom

> Then they - and others - are and have been welcome to clearly express on the
> NVO3 mailing list why those objections should be ignored.   That is an
> discussed in the WG, before the design begins. Having it as a separate draft
> might help. If then you want to merge it or morph it into the design doc
> that will be fine.
>
>
> confirmation or disagreement on interest in creating a DP encapsulation that
> addresses the various technical concerns".
>
> Thanks,
> Fabio
>
>
>
>> of Fabio's note.  Assuming we must go there, having a section that documents