Subject: [nvo3] Draft Geneve

From: "Anton Ivanov (antivano)" <[email-masked]>
Date: Fri, 28 Feb 2014 08:30:15 +0000

Hi all,

I just finished reading this draft.

I am going to ignore the caption under the diagram at this point - it
does not match to the diagram and neither does the rest of the draft (we
will come back to it later).

So, first and foremost, the diagram. I actually really like it. It is
very close to what we have been showing to customers as a product
prototype and will be shipping in the near future. Namely - I like the
presence of the physical(s) in the top right corner. Physicals are an
integral part of a datacenter and need to talk to the overlay too.

It is quite sad that after getting this right, the draft cripples it by
limiting the scope and failing to address what is actually shown on that
diagram.

In order for physicals to talk to the overlay they need to be able to
connect to the overlay. In real life this means well known and well
supported encapsulations - EoGRE, EoL2TPv3 and more recently VXLAN. The
former two are supported on most "hosts" as well as nearly all network
equipment. The latter is getting there too. A new and wonderful
encapsulation which on top of all is not directly supported and mandates
me to run a virtual switch everywhere is not a "well supported
encapsulation". It does not fit the bill and does not match the diagram.

There are two ways to approach this from here onwards. One is to
continue the critique and discussion. IMHO this is pointless and
violates the spirit of IETF: "We believe in rough consensus and working
code".

The consensus is there - most of the networking world uses either EoGRE
or EoL2TPv3 (VXLAN has its established use cases too). These are
established standards with RFCs. We will now _ADD_ _THE_ _CODE_ so we
can get out of this "I'd like to invent a new encapsulation today"
madness once and for all.

    * We just contributed a GPL2 EoL2TPv3 direct support for qemu-kvm to
the qemu project.
    * We also made a binding promise to offer equivalent code to UML (we
have the code there, just the patch-sets are too big to submit at once).
    * Containers can talk EoL2TPv3 directly already.

All of these can use this to talk directly VM to VM, Physical to VM and
Switch to VM (as on the diagram).

It is fully supported for switching - you simply configure a EoL2TPv3
interface on your local host and bring it into OVS, Bridge or other
local switching system of your choice. You can also switch remotely -
the NVE is no longer mandatory co-hosted on the same machine. So if you
want to do all of your datacenter switching on a "big router/switch" you
can now do so. All of them can talk the encaps, most have support for
running thousands (some up to tens and hundreds of thousands) of NVEs too.

You also can talk to host - just configure a tunnel "to yourself". Most
importantly you can talk VM to VM and VM to physical router as needed
for service chaining.

This brings me back to the caption under the diagram. That caption is
complete nonsense. You can implement means of communicating between all
the elements on it without a mandatory switch. Why on earth do you need
to cripple the spec and specify it after that?

I am aware that EoLT2Pv3 is not a candidate for NVO3 at present. We will
address that by contribute equivalent GRE code at a later date (next
couple months).

In fact, anyone can take the driver we contributed and make it support
GRE, Geneve or carrier pigeons. This is because this contribution has
produced an interesting side effect - anyone who would like to implement
in software a new encaps that can be presented as a socket interface on
a host running Type 2 environment can now do so in an afternoon or less.
It is no longer "state of the art" activity. The barrier to entry and
the technical merit of "new encaps" if it is only an encaps (no control
plane component) has just been lowered to somewhere around zero. It is
now "boring".

This is inclusive of Geneve itself. As it has no other merit besides
being new encaps I do not see why it should be left to live under these
circumstances. I'd rather have people include the remaining key encaps
out there - L2TPv3 and concentrate on getting the control plane portion
working.

Some final clarifications:

1. Drafts - L2TPv3 is an RFC and has been around for many years. There
is no need for extra drafts.
2. IPR - the same mechanics for overlay termination have been used in
storage for many years providing a plethora of prior art. Similarly,
non-standard versions of the same approach (VDE, socket,etc) have been
around since the dawn of qemu and UML. There is nothing to patent here.

A

---

From: Brad Hedlund <[email-masked]>
Date: Fri, 28 Feb 2014 06:14:01 -0800 (PST)

The diagram in the Geneve draft depicts tunnels terminating on both virtual switches in a Hypervisor, as well as physical switches connecting to non-virtual physical hosts.
As such, the later brings the physical hosts into the overlay without any modification to the software stack on the physical hosts.
This is already happening today with VXLAN, with commercial switching silicon providing line rate VXLAN encap/decap.  The physical switch vendor can then provide interoperability with a third party control plane that can provision tunnels between the virtual and physical switches.

There's nothing preventing the physical hosts from implementing Geneve tunnels in their own software stack and bypassing the need for terminating tunnels on a physical switch.
The diagram just depicts what the authors felt would be most common deployment scenario.

As I see it, the point with Geneve is that the tunneling protocol is actually the least interesting part of the overall system.  So perhaps the industry can work with a common tunneling format that provides both data plane interoperability and sufficient room in the header for implementation specific differentiation (metadata).  Instead, perhaps the industry can focus on competing and differentiating at the control and management planes.  Not too dissimilar from what happened when we agreed on Ethernet and IP encapsulation.

Cheers,
Brad

----- Original Message -----
From: "Anton Ivanov (antivano)" <[email-masked]>
To: [email-masked]
Sent: Friday, February 28, 2014 2:30:15 AM
Subject: [nvo3] Draft Geneve

Hi all,

A

_______________________________________________
nvo3 mailing list
[email-masked]
https://urldefense.proofpoint.com/v1/url?u=https://www.ietf.org/mailman/listinfo/nvo3&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=LfVwmzZtwW32snSTUvjgMl4EgksMaIQ6nBSzwpJYhxw%3D%0A&m=HIV6ylhSIvlBLF2JcY4AbvKIp4AytOoWy1TnXn9PrCc%3D%0A&s=4df281d1fecc67d941f33a355364c2e128de78da24a74dfa6af38f8f85e090d2

---

From: "Anton Ivanov (antivano)" <[email-masked]>
Date: Fri, 28 Feb 2014 14:40:51 +0000

> This is already happening today with VXLAN, with commercial switching silicon providing line rate VXLAN encap/decap.  The physical switch vendor can then provide interoperability with a third party…

Correct. I am not questioning that.

My point is that in the case of L2TPv3 and GRE it _HAS_ happened. Many
years ago. Different platforms though (BNG, Carrier systems, etc).

Also your VXLAN interpretation still means that the host does not
participate in the overlay. It is mediated by an off-host switch through
an interim step as raw Ethernet and you just added a silicon requirement
for that switch as well.

This is not the host participating in the overlay. There is a benefit
for the host understanding the overlay.

Trivial example - if the host can understand the overlay it can present
an overlay endpoint as an ethernet interface to a lightweight container.
No overhead. Direct overlay to visualization instance. Thousands of
instances on a host too if you want to.

> As I see it, the point with Geneve is that the tunneling protocol is actually the least interesting part of the overall system. 

We are in complete agreement. However I am taking this one step further.
It is not just the least interesting part - it has already been done. It
is in the stack on most OSes. It has been in the stack on most OSes for
5 years running. There is no reason whatsoever to reinvent the wheel.

The protocols are there, present, usable and should be used.

>  So perhaps the industry can work with a common tunneling format that provides both data plane interoperability and sufficient room in the header for implementation specific differentiation (metadata).  

L2TPv3 has 32 bits of what can be used as VNID (session). I do not see
how a proposal with a 24 bit VNID space has any merit over that. If 24
bits are enough, there should be a valid argument on how suddenly 32 bit
have become smaller than 24.

> Instead, perhaps the industry can focus on competing and differentiating at the control and management planes.  Not too dissimilar from what happened when we agreed on Ethernet and IP encapsulation.

Again - completely agree. In between themselves VXLAN, L2TPv3 and GRE
cover all key encaps basics and are supported on all platforms of
interest. There is no need for yet another unoriginal encapsulation.

A.

---

From: Brad Hedlund <[email-masked]>
Date: Fri, 28 Feb 2014 07:08:56 -0800 (PST)

The Geneve draft proposes a 24-bit VNI, in addition to 32-bits of "Variable Length Options".
Those later 32 bits can be used entirely by the system designer for carrying implementation specific metadata, such as for example carrying service chaining relevant information from one middle-box to the next.

Cheers,
Brad 

----- Original Message -----
From: "Anton Ivanov (antivano)" <[email-masked]>
To: "Brad Hedlund" <[email-masked]>
Cc: [email-masked]
Sent: Friday, February 28, 2014 8:40:51 AM
Subject: Re: [nvo3] Draft Geneve

A.

---

From: Pankaj Garg <[email-masked]>
Date: Fri, 28 Feb 2014 15:30:16 +0000

Yes, the key differentiator in Geneve is to allow multiple variable length options in the unit of 32-bits, _without_ breaking hardware offloads that are critical for software endpoint performance. These options can be standard options or vendor specific, to allow vendors to evolve their data plane and experiment/innovate using meta-data, and standardize the useful ones at a later point.

Thanks,
Pankaj

-----Original Message-----
From: nvo3 [mailto:[email-masked]] On Behalf Of Brad Hedlund
Sent: Friday, February 28, 2014 8:39 PM
To: Anton Ivanov (antivano)
Cc: [email-masked]
Subject: Re: [nvo3] Draft Geneve

Cheers,
Brad 

----- Original Message -----
From: "Anton Ivanov (antivano)" <[email-masked]>
To: "Brad Hedlund" <[email-masked]>
Cc: [email-masked]
Sent: Friday, February 28, 2014 8:40:51 AM
Subject: Re: [nvo3] Draft Geneve

My point is that in the case of L2TPv3 and GRE it _HAS_ happened. Many years ago. Different platforms though (BNG, Carrier systems, etc).

Also your VXLAN interpretation still means that the host does not participate in the overlay. It is mediated by an off-host switch through an interim step as raw Ethernet and you just added a silicon requirement for that switch as well.

This is not the host participating in the overlay. There is a benefit for the host understanding the overlay.

Trivial example - if the host can understand the overlay it can present an overlay endpoint as an ethernet interface to a lightweight container.
No overhead. Direct overlay to visualization instance. Thousands of instances on a host too if you want to.

We are in complete agreement. However I am taking this one step further.
It is not just the least interesting part - it has already been done. It is in the stack on most OSes. It has been in the stack on most OSes for
5 years running. There is no reason whatsoever to reinvent the wheel.

>  So perhaps the industry can work with a common tunneling format that provides both data plane interoperability and sufficient room in the header for implementation specific differentiation (metada…

L2TPv3 has 32 bits of what can be used as VNID (session). I do not see how a proposal with a 24 bit VNID space has any merit over that. If 24 bits are enough, there should be a valid argument on how suddenly 32 bit have become smaller than 24.

> Instead, perhaps the industry can focus on competing and differentiating at the control and management planes.  Not too dissimilar from what happened when we agreed on Ethernet and IP encapsulation.

Again - completely agree. In between themselves VXLAN, L2TPv3 and GRE cover all key encaps basics and are supported on all platforms of interest. There is no need for yet another unoriginal encapsulation.

A.

> does not match to the diagram and neither does the rest of the draft 
> (we will come back to it later).
>
> It is quite sad that after getting this right, the draft cripples it 
> by limiting the scope and failing to address what is actually shown on 
> that diagram.
>
> supported encapsulations - EoGRE, EoL2TPv3 and more recently VXLAN. 
> The former two are supported on most "hosts" as well as nearly all 
> network equipment. The latter is getting there too. A new and 
> wonderful encapsulation which on top of all is not directly supported 
> and mandates me to run a virtual switch everywhere is not a "well 
> supported encapsulation". It does not fit the bill and does not match the diagram.
>
> violates the spirit of IETF: "We believe in rough consensus and 
> working code".
>
> The consensus is there - most of the networking world uses either 
> EoGRE or EoL2TPv3 (VXLAN has its established use cases too). These are 
>     * We just contributed a GPL2 EoL2TPv3 direct support for qemu-kvm 
> to the qemu project.
>     * We also made a binding promise to offer equivalent code to UML 
> (we have the code there, just the patch-sets are too big to submit at once).
> All of these can use this to talk directly VM to VM, Physical to VM 
> and Switch to VM (as on the diagram).
>
> the NVE is no longer mandatory co-hosted on the same machine. So if 
> you want to do all of your datacenter switching on a "big 
> router/switch" you can now do so. All of them can talk the encaps, 
> most have support for running thousands (some up to tens and hundreds of thousands) of NVEs too.
>
> You also can talk to host - just configure a tunnel "to yourself". 
> Most importantly you can talk VM to VM and VM to physical router as 
> needed for service chaining.
>
> complete nonsense. You can implement means of communicating between 
> all the elements on it without a mandatory switch. Why on earth do you 
> need to cripple the spec and specify it after that?
>
> I am aware that EoLT2Pv3 is not a candidate for NVO3 at present. We 
> will address that by contribute equivalent GRE code at a later date 
> (next couple months).
>
> produced an interesting side effect - anyone who would like to 
> implement in software a new encaps that can be presented as a socket 
> interface on a host running Type 2 environment can now do so in an afternoon or less.
> the technical merit of "new encaps" if it is only an encaps (no 
> control plane component) has just been lowered to somewhere around 
> zero. It is now "boring".
>
> being new encaps I do not see why it should be left to live under 
> these circumstances. I'd rather have people include the remaining key 
> encaps out there - L2TPv3 and concentrate on getting the control plane 
> portion working.
>
> https://urldefense.proofpoint.com/v1/url?u=https://www.ietf.org/mailma
> n/listinfo/nvo3&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=LfVwmzZtwW32snST
> UvjgMl4EgksMaIQ6nBSzwpJYhxw%3D%0A&m=HIV6ylhSIvlBLF2JcY4AbvKIp4AytOoWy1
> TnXn9PrCc%3D%0A&s=4df281d1fecc67d941f33a355364c2e128de78da24a74dfa6af3
> 8f8f85e090d2

_______________________________________________
nvo3 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/nvo3

---

From: "Anton Ivanov (antivano)" <[email-masked]>
Date: Fri, 28 Feb 2014 15:34:41 +0000

> Those later 32 bits can be used entirely by the system designer for carrying implementation specific metadata, such as for example carrying service chaining relevant information from one middle-box…

L2TPv3 - 32 bit VNI

While the standard itself does not specify this, most implementations
allow an extra OFFSET in the configuration so you can add metadata
fields (or eat fields you do not like). We can standardize it to be
whatever number of bytes we would want it to be for NVO3 use. If the
consensus is that it should be 32 bit, then it may be better if we
specify it as 64 bit (for alignment purposes).

Implementations that do not use it simply need to be configured to skip
over it and will operate in compatibility mode. I simply set offset in
the command to be equal to the number of bits reserved for metadata.
Well supported feature across multiple platforms.

Once again, I can achieve the same using what is already there. I do not
need to invent an "encaps du jour" just for this. It has already been
done. Many times.

Example from Linux (I could have used the asr9k or nearly any other box
on the market for an example here):

Usage: ip l2tp add tunnel
          remote ADDR local ADDR
          tunnel_id ID peer_tunnel_id ID
          [ encap { ip | udp } ]
          [ udp_sport PORT ] [ udp_dport PORT ]
Usage: ip l2tp add session [ name NAME ]
          tunnel_id ID
          session_id ID peer_session_id ID
          [ cookie HEXSTR ] [ peer_cookie HEXSTR ]
          [ offset OFFSET ] [ peer_offset OFFSET ]
          [ l2spec_type L2SPEC ]
       ip l2tp del tunnel tunnel_id ID
       ip l2tp del session tunnel_id ID session_id ID
       ip l2tp show tunnel [ tunnel_id ID ]
       ip l2tp show session [ tunnel_id ID ] [ session_id ID ]

Where: NAME   := STRING
       ADDR   := { IP_ADDRESS | any }
       PORT   := { 0..65535 }
       ID     := { 1..4294967295 }
       HEXSTR := { 8 or 16 hex digits (4 / 8 bytes) }

A.

---

From: "Anton Ivanov (antivano)" <[email-masked]>
Date: Fri, 28 Feb 2014 15:47:03 +0000

> Yes, the key differentiator in Geneve is to allow multiple variable length options in the unit of 32-bits, _without_ breaking hardware offloads that are critical for software endpoint performance.…

So how is that different from me specifying a different offset for data
in l2tpv3 in multiples of 32 bits? We just specify what the meanings of
those should be.

The whole offload section of draft-geneve can be applied there to same
effect. The only difference compared to an existing published and well
supported RFC is that we have to standardize the way the configurable
offset is handled.

A.

---

From: "Ken Gray (kegray)" <[email-masked]>
Date: Fri, 28 Feb 2014 16:18:18 +0000

That work (chaining and metadata) is already being addressed in the SFC
workgroup by charter.  At the very least, the draft should move if that's
it's primary thrust.  In that work, there is no coupling between the
header for forwarding in an overlay and the header for chaining and
passing metadata Šand I think that was on purpose.

On 2/28/14 10:08 AM, "Brad Hedlund" <[email-masked]> wrote:

>Those later 32 bits can be used entirely by the system designer for
>carrying implementation specific metadata, such as for example carrying
>service chaining relevant information from one middle-box to the next.
>
>Cheers,
>Brad 
>
>
>> The diagram in the Geneve draft depicts tunnels terminating on both
>>virtual switches in a Hypervisor, as well as physical switches
>>connecting to non-virtual physical hosts.
>> As such, the later brings the physical hosts into the overlay without
>>any modification to the software stack on the physical hosts.
>> This is already happening today with VXLAN, with commercial switching
>>silicon providing line rate VXLAN encap/decap.  The physical switch
>>vendor can then provide interoperability with a third party control
>>plane that can provision tunnels between the virtual and physical
>>switches.
>
>> There's nothing preventing the physical hosts from implementing Geneve
>>tunnels in their own software stack and bypassing the need for
>>terminating tunnels on a physical switch.
>> As I see it, the point with Geneve is that the tunneling protocol is
>>actually the least interesting part of the overall system.
>
>>  So perhaps the industry can work with a common tunneling format that
>>provides both data plane interoperability and sufficient room in the
>>header for implementation specific differentiation (metadata).
>
>> Instead, perhaps the industry can focus on competing and
>>differentiating at the control and management planes.  Not too
>>dissimilar from what happened when we agreed on Ethernet and IP
>>encapsulation.
>
>>https://urldefense.proofpoint.com/v1/url?u=https://www.ietf.org/mailman/l
>>istinfo/nvo3&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=LfVwmzZtwW32snSTUvjgMl
>>4EgksMaIQ6nBSzwpJYhxw%3D%0A&m=HIV6ylhSIvlBLF2JcY4AbvKIp4AytOoWy1TnXn9PrCc
>>%3D%0A&s=4df281d1fecc67d941f33a355364c2e128de78da24a74dfa6af38f8f85e090d2
>

---

From: Pankaj Garg <[email-masked]>
Date: Fri, 28 Feb 2014 16:26:42 +0000

Geneve options are not specific to service chaining, though a service chain blob can be one of the option in Geneve header. So it does belongs to network virtualization encapsulation format and not to SFC charter.

-----Original Message-----
From: nvo3 [mailto:[email-masked]] On Behalf Of Ken Gray (kegray)
Sent: Friday, February 28, 2014 9:48 PM
To: Brad Hedlund; Anton Ivanov (antivano)
Cc: [email-masked]
Subject: Re: [nvo3] Draft Geneve

That work (chaining and metadata) is already being addressed in the SFC workgroup by charter.  At the very least, the draft should move if that's it's primary thrust.  In that work, there is no coupling between the header for forwarding in an overlay and the header for chaining and passing metadata Šand I think that was on purpose.

On 2/28/14 10:08 AM, "Brad Hedlund" <[email-masked]> wrote:

>participate in the overlay. It is mediated by an off-host switch 
>through an interim step as raw Ethernet and you just added a silicon 
>requirement for that switch as well.
>
>> There's nothing preventing the physical hosts from implementing 
>>Geneve tunnels in their own software stack and bypassing the need for 
>It is not just the least interesting part - it has already been done. 
>It is in the stack on most OSes. It has been in the stack on most OSes 
>for
>bits are enough, there should be a valid argument on how suddenly 32 
>bit have become smaller than 24.
>
>> prototype and will be shipping in the near future. Namely - I like 
>> the presence of the physical(s) in the top right corner. Physicals 
>> are an integral part of a datacenter and need to talk to the overlay too.
>>
>> by limiting the scope and failing to address what is actually shown 
>> on that diagram.
>>
>> EoGRE or EoL2TPv3 (VXLAN has its established use cases too). These 
>> are established standards with RFCs. We will now _ADD_ _THE_ _CODE_ 
>> so we can get out of this "I'd like to invent a new encapsulation today"
>> all the elements on it without a mandatory switch. Why on earth do 
>> you need to cripple the spec and specify it after that?
>>
>> In fact, anyone can take the driver we contributed and make it 
>> support GRE, Geneve or carrier pigeons. This is because this 
>> contribution has produced an interesting side effect - anyone who 
>> would like to implement in software a new encaps that can be 
>> presented as a socket interface on a host running Type 2 environment can now do so in an afternoon or less.
>> encaps out there - L2TPv3 and concentrate on getting the control 
>> plane portion working.
>>
>> 1. Drafts - L2TPv3 is an RFC and has been around for many years. 
>> There is no need for extra drafts.
>> non-standard versions of the same approach (VDE, socket,etc) have 
>> been around since the dawn of qemu and UML. There is nothing to patent here.
>>
>> A
>>
>>n/l

---

From: "Anton Ivanov (antivano)" <[email-masked]>
Date: Fri, 28 Feb 2014 16:49:52 +0000

> Geneve options are not specific to service chaining, though a service chain blob can be one of the option in Geneve header. So it does belongs to network virtualization encapsulation format and not…

Disagree.

>From an encapsulation perspective it is a reduced form of what L2TPv3
static tunnels can do already:

1. Smaller VNID
2. No cookies so no defence against misconfiguration
3. Unknown security interactions and no analysis of interactions with
IPSEC so no clarity on extending beyond datacenter.
4. While offset in L2TPv3 is not part of the standard, it is a most
common implementation feature allowing for same "metadata" blob
behaviour. Specifying that blob to hit an aligned address so you can
configure offload for it is trivial. It is not worth it to waste a draft
(and a potential RFC) on what is an implementation detail. What if I
implement an offload engine which works on unaligned addresses? Go and
change the draft just because of that?
5. No silicon implementation, no widely deployed software implementations.

So if it is "just an encapsulation" there is no reason for it to exist -
it has all been done, we should just standardize the existing offset
feature and be done with it.

I agree with Brad that the encaps part is the least interesting bit -
the key here is the way this is treated in a system - the meaning of
life, universe and the metadata blob. That at the moment is being looked
at only in SFC, so it does indeed belong there despite the fact that
they are looking at a reduced set of meanings for that.

A.

---

From: Pankaj Garg <[email-masked]>
Date: Fri, 28 Feb 2014 16:57:34 +0000

We can discuss merits of Geneve vs VXLAN vs NVGRE vs STT vs L2TPV3 vs <Name your favorite protocol> when it comes to standardization for network virtualization. My point was that Geneve is designed for network virtualization and not only for service chaining. We have many use cases of ability to carry meta-data to evolve network virtualization that have nothing to do with service chaining and hence the right discussion forum for Geneve is NVO3 (IMHO).

-----Original Message-----
From: Anton Ivanov (antivano) [mailto:[email-masked]] 
Sent: Friday, February 28, 2014 10:20 PM
To: Pankaj Garg
Cc: Ken Gray (kegray); Brad Hedlund; [email-masked]
Subject: Re: [nvo3] Draft Geneve

Disagree.

>From an encapsulation perspective it is a reduced form of what L2TPv3 static tunnels can do already:

1. Smaller VNID
2. No cookies so no defence against misconfiguration 3. Unknown security interactions and no analysis of interactions with IPSEC so no clarity on extending beyond datacenter.
4. While offset in L2TPv3 is not part of the standard, it is a most common implementation feature allowing for same "metadata" blob behaviour. Specifying that blob to hit an aligned address so you can configure offload for it is trivial. It is not worth it to waste a draft (and a potential RFC) on what is an implementation detail. What if I implement an offload engine which works on unaligned addresses? Go and change the draft just because of that?
5. No silicon implementation, no widely deployed software implementations.

So if it is "just an encapsulation" there is no reason for it to exist - it has all been done, we should just standardize the existing offset feature and be done with it.

I agree with Brad that the encaps part is the least interesting bit - the key here is the way this is treated in a system - the meaning of life, universe and the metadata blob. That at the moment is being looked at only in SFC, so it does indeed belong there despite the fact that they are looking at a reduced set of meanings for that.

A.

>> carrying implementation specific metadata, such as for example 
>> carrying service chaining relevant information from one middle-box to the next.
>>
>> Cheers,
>> Brad
>>
>>
>>> As such, the later brings the physical hosts into the overlay 
>>> without any modification to the software stack on the physical hosts.
>>> This is already happening today with VXLAN, with commercial 
>>> switching silicon providing line rate VXLAN encap/decap.  The 
>>> physical switch vendor can then provide interoperability with a 
>>> third party control plane that can provision tunnels between the 
>>> virtual and physical switches.
>> My point is that in the case of L2TPv3 and GRE it _HAS_ happened. 
>> Many years ago. Different platforms though (BNG, Carrier systems, etc).
>>
>> Trivial example - if the host can understand the overlay it can 
>> present an overlay endpoint as an ethernet interface to a lightweight container.
>>> Geneve tunnels in their own software stack and bypassing the need 
>>> for terminating tunnels on a physical switch.
>> It is in the stack on most OSes. It has been in the stack on most 
>> OSes for
>>>  So perhaps the industry can work with a common tunneling format 
>>> that provides both data plane interoperability and sufficient room 
>>> in the header for implementation specific differentiation (metadata).
>> L2TPv3 has 32 bits of what can be used as VNID (session). I do not 
>> see how a proposal with a 24 bit VNID space has any merit over that. 
>> If 24 bits are enough, there should be a valid argument on how 
>> suddenly 32 bit have become smaller than 24.
>>
>>> I am going to ignore the caption under the diagram at this point - 
>>> it does not match to the diagram and neither does the rest of the 
>>> draft (we will come back to it later).
>>>
>>> So, first and foremost, the diagram. I actually really like it. It 
>>> is very close to what we have been showing to customers as a product 
>>> In order for physicals to talk to the overlay they need to be able 
>>> to connect to the overlay. In real life this means well known and 
>>> well supported encapsulations - EoGRE, EoL2TPv3 and more recently VXLAN.
>>> wonderful encapsulation which on top of all is not directly 
>>> supported and mandates me to run a virtual switch everywhere is not 
>>> a "well supported encapsulation". It does not fit the bill and does not match the diagram.
>>>
>>>     * We just contributed a GPL2 EoL2TPv3 direct support for 
>>> qemu-kvm to the qemu project.
>>> It is fully supported for switching - you simply configure a 
>>> EoL2TPv3 interface on your local host and bring it into OVS, Bridge 
>>> or other local switching system of your choice. You can also switch 
>>> remotely - the NVE is no longer mandatory co-hosted on the same 
>>> machine. So if you  want to do all of your datacenter switching on a 
>>> "big router/switch" you  can now do so. All of them can talk the 
>>> encaps, most have support for  running thousands (some up to tens 
>>> and hundreds of thousands) of NVEs too.
>>>
>>> This brings me back to the caption under the diagram. That caption 
>>> is complete nonsense. You can implement means of communicating 
>>> between all the elements on it without a mandatory switch. Why on 
>>> earth do you need to cripple the spec and specify it after that?
>>>
>>> It is no longer "state of the art" activity. The barrier to entry 
>>> and the technical merit of "new encaps" if it is only an encaps (no 
>>> these circumstances. I'd rather have people include the remaining 
>>> key encaps out there - L2TPv3 and concentrate on getting the control 
>>> plane portion working.
>>>
>>> 2. IPR - the same mechanics for overlay termination have been used 
>>> in storage for many years providing a plethora of prior art. 
>>> Similarly, non-standard versions of the same approach (VDE, 
>>> socket,etc) have been around since the dawn of qemu and UML. There is nothing to patent here.
>>>
>>> A
>>>
>>> istinfo/nvo3&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=LfVwmzZtwW32snSTU
>>> vj
>>> gMl
>>> 4EgksMaIQ6nBSzwpJYhxw%3D%0A&m=HIV6ylhSIvlBLF2JcY4AbvKIp4AytOoWy1TnXn
>>> 9P
>>> rCc
>>> %3D%0A&s=4df281d1fecc67d941f33a355364c2e128de78da24a74dfa6af38f8f85e
>>> 09
>>> 0d2

---

From: Tom Herbert <[email-masked]>
Date: Fri, 28 Feb 2014 09:45:57 -0800

Perhaps some perspectives from a customer with a large scale
deployment of network virtualization (Google Compute Engine) might be
helpful.

- Design of encapsulation protocol is *very*important which might be
contrary to prevailing opinion. This is true because of two reasons:
1) we need interoperability with HW optimizations or sacrifice
performance 2) this is directly tied to how we implement security and
control.
- Control plane as a standard is honestly less interesting. We are not
building a new network, so we're already integrating into an existing
routing and TE control plane. Interoperability is much less of a
concern in a control plane for us. If a standard is defined which is
demonstrably better, we'll happily use it, but I don't see ever being
compelled to use a standard in this case.
- Protocol design must be separate from the product. We need an
encapsulation that does not require HW support, is generic, and
extensible for existing uses and those beyond what we currently
conceive.
- We need a generic encapsulation protocol not just for network
virtualization, but one which will be ubiquitous within the data
center. My preference is that we only have two protocols on the wire:
TCP and UDP. This maximizes HW interoperability and supports
homogeneity.
- Extensibility is a big requirement for us. For our current
deployment we needed additional security data in each packet, the
inability to add any fields to vxlan and nvgre make them nonstarters.
We will likely have other needs for extensibility, we also need the
ability to have completely proprietary options.
- As for the concerns of hardware vendors that variable length headers
are difficult to implement, I sympathize. But, I believe there are
ways to mitigate this. Data centers tend to be homogenous environments
so that among the possibilities of VLH header formats only one or
maybe two make up the vast majority of packets and hence are what HW
can optimize for. Changes to the common format are rare events,
however they will occur and it's *not* acceptable that we need to
change out HW for that.
- We are *not* trying to replace vlan with something better. I believe
most large data centers are L3 routed anyway, so there's no universal
need for a migration or backwards compatibility.
- Similarly, we are not trying to solve a an L2 encapsulation problem.
The vast majority of our customers are using IPv4 or IPv6, we can
encapsulate these directly on an overlay network.  I really don't want
to have to wrap everything in a dummy Ethernet frame and take a 1% tax
on all my packets just for that. One reason why we don't use l2tp.
- Security is also a critical requirements area. We need an
encapsulation which allows both E2E security (e.g. IPsec) as well as
firewalling in networking (e.g. firewalling based on vni). Note, I am
not distinguishing security requirements as being intra versus
inter/extra data center. We really need a path where we can secure
every packet if needed, and have multiple levels of security (once we
allow 3rd party VMs in the network idea of simple security perimeters
goes out the window!). The security story for l2tp, geneve, etc. seem
to be use IPsec, but I don't think that is sufficient, as we lose the
benefits of having encap in outer headers (for hash computation,
firewall).
- Related to the above, this is a major reason why GUE uses IP
protocol instead of Ethertype as next protocol. This allows as
protocols stack like IP-UDP-GUE-ESP-packet. So we can encap ESP
directly. I don't see a way to do this with Ethertype.
- Control and QoS (like CC, rate limiting, etc.) are important. While
we can do a lot with external out of band QoS mechanisms, congestion
control within the encap protocols does seem desirable for a portion
of traffic (that which has non-conformant implementation). A point for
l2tp.
- Scaling is important too. I'm surprised there's any argument in
making vni's at least 32 bits, AFAICT 24 bits is just an artifact of
nvgre in overloading keyid with vni and flow id. In any case, we've
already deployed a 32 bit vni's and I really doubt we'd ever purposely
reduce the space. It is possible that we might need a larger space
some day (e.g. 64 bits), but that is another case in extensibility.

Tom

---

From: "Anton Ivanov (antivano)" <[email-masked]>
Date: Fri, 28 Feb 2014 17:47:13 +0000

On 28/02/14 16:57, Pankaj Garg wrote:
> We can discuss merits of Geneve vs VXLAN vs NVGRE vs STT vs L2TPV3 vs <Name your favorite protocol> when it comes to standardization for network virtualization. 

I already said what I had to say here. There is no discussion to be had.

> My point was that Geneve is designed for network virtualization and not only for service chaining. We have many use cases of ability to carry meta-data to evolve network virtualization that have nothing to do with service chaining and hence the right discussion forum for Geneve is NVO3 (IMHO).

You again missed Kens and my point.

We do not dispute the need to carry metadata and this is exactly the
discussion we should be having.

However the need to carry metadata has to be _DECOUPLED_ from
encapsulation requirements.

That is trivial.

Use case - I have a Geneve  (+ metadata) on the left, I cross an NVI and
I exit on the right via let's say VXLAN. If metadata is naturally welded
to encapsulation as you propose this means losing all of it.

Sorry, this does not fly. You asking the IETF under the guise of
standard to grant you monopoly on any deployment where you got a leg in.

Errr.. that shall be a no then :)

A.

---

From: Pankaj Garg <[email-masked]>
Date: Fri, 28 Feb 2014 18:28:12 +0000

The definition of meta-data for a feature should come from their respective WG, e.g. service-chaining blob design should come from SFC. How, the meta-data data is carried in an encapsulation comes from the design of the encapsulation format. The point of Geneve is that it would allow the meta-data to be carried without breaking NIC hardware offloads, whereas VXLAN, NVGRE etc. would require NIC hardware changes to carry meta-data. 

-----Original Message-----
From: nvo3 [mailto:[email-masked]] On Behalf Of Anton Ivanov (antivano)
Sent: Friday, February 28, 2014 11:17 PM
To: [email-masked]
Subject: Re: [nvo3] Draft Geneve

We do not dispute the need to carry metadata and this is exactly the discussion we should be having.

However the need to carry metadata has to be _DECOUPLED_ from encapsulation requirements.

That is trivial.

Use case - I have a Geneve  (+ metadata) on the left, I cross an NVI and I exit on the right via let's say VXLAN. If metadata is naturally welded to encapsulation as you propose this means losing all of it.

Sorry, this does not fly. You asking the IETF under the guise of standard to grant you monopoly on any deployment where you got a leg in.

A.

>>> This is not the host participating in the overlay. There is a 
>>> benefit for the host understanding the overlay.
>>>
>>>> As I see it, the point with Geneve is that the tunneling protocol 
>>>> is actually the least interesting part of the overall system.
>>> Again - completely agree. In between themselves VXLAN, L2TPv3 and 
>>> GRE cover all key encaps basics and are supported on all platforms 
>>> of interest. There is no need for yet another unoriginal encapsulation.
>>>
>>> A.
>>>
>>>> Cheers,
>>>> Brad
>>>>
>>>>
>>>> is very close to what we have been showing to customers as a 
>>>> product prototype and will be shipping in the near future. Namely - 
>>>> I like the presence of the physical(s) in the top right corner. 
>>>> Physicals are an integral part of a datacenter and need to talk to the overlay too.
>>>>
>>>> It is quite sad that after getting this right, the draft cripples 
>>>> it by limiting the scope and failing to address what is actually 
>>>> shown on that diagram.
>>>>
>>>>     * We also made a binding promise to offer equivalent code to 
>>>> UML (we have the code there, just the patch-sets are too big to submit at once).
>>>> machine. So if you  want to do all of your datacenter switching on 
>>>> a "big router/switch" you  can now do so. All of them can talk the 
>>>> This is inclusive of Geneve itself. As it has no other merit 
>>>> besides being new encaps I do not see why it should be left to live 
>>>> under these circumstances. I'd rather have people include the 
>>>> remaining key encaps out there - L2TPv3 and concentrate on getting 
>>>> the control plane portion working.
>>>>
>>>> https://urldefense.proofpoint.com/v1/url?u=https://www.ietf.org/mai
>>>> l
>>>> ma
>>>> n/l

---

From: Dino Farinacci <[email-masked]>
Date: Fri, 28 Feb 2014 10:36:09 -0800

> As I see it, the point with Geneve is that the tunneling protocol is actually the least interesting part of the overall system.  So perhaps the industry can work with a common tunneling format that…

Then why did you create yet another encapsulation the world does not need?

Dino

P.S. I know the answer.  ;-)

---

From: "Anton Ivanov (antivano)" <[email-masked]>
Date: Fri, 28 Feb 2014 19:00:11 +0000

> The definition of meta-data for a feature should come from their respective WG, e.g. service-chaining blob design should come from SFC. How, the meta-data data is carried in an encapsulation comes…

I have read that section about 10 times and I fail to see you doing
anything else beside putting it on 32 bit boundary and mandating it ends
on a 32 bit boundary.

If that is the case, that can be done with any other encaps. No need to
define a new one - just insert metadata in front of payload and pad to
boundaries if needed.

If that is not the case, can you please expand on exactly what it makes
it so special in the draft so that an insertion of a metadata blob at 32
bit boundary and ending on 32 bit boundary into L2TPv3, GRE, VXLAN, etc
does not fulfill that. It is not evident from the draft in its current form.

A.

---

From: Sam Aldrin <[email-masked]>
Date: Fri, 28 Feb 2014 15:24:12 -0800

Hi all,

Read the draft but have few questions on the same line others have asked.

- Is this draft intended for standardizing within NVo3 WG? The status
indicates it as informational. Also it is good to have it as
draft-nvo3...., if it is meant for NVo3 WG.
- I fail to find good reasoning, in the current version of the draft, on
why design of encap transport header should be closely associated with
metadata OR closely tied together? Could you add more details to clarify?
- Has the GAP analysis draft concluded on the assertion you made in your
draft, for ex:
     "Existing tunnel protocols have each attempted to solve different
aspects of these new requirements, only to be quickly rendered out of date
by changing control plane implementations and advancements."
- In the past WG sessions and over the emails, the indication given was
that various encap technologies already exist. Defining a good control
plane technology is the need for the WG, no?
- Finally, would like to see more details on why new encap is needed and
could solve the problem, while extensions to VXLAN or NVGRE or LISP cannot
solve.

thanks
-sam

On Fri, Feb 28, 2014 at 11:00 AM, Anton Ivanov (antivano) <
[email-masked]> wrote:

> > The definition of meta-data for a feature should come from their
> respective WG, e.g. service-chaining blob design should come from SFC. How,
> the meta-data data is carried in an encapsulation comes from the design of
> the encapsulation format. The point of Geneve is that it would allow the
> meta-data to be carried without breaking NIC hardware offloads, whereas
> VXLAN, NVGRE etc. would require NIC hardware changes to carry meta-data.
>
> >> We can discuss merits of Geneve vs VXLAN vs NVGRE vs STT vs L2TPV3 vs
> <Name your favorite protocol> when it comes to standardization for network
> virtualization.
> >> My point was that Geneve is designed for network virtualization and not
> only for service chaining. We have many use cases of ability to carry
> meta-data to evolve network virtualization that have nothing to do with
> service chaining and hence the right discussion forum for Geneve is NVO3
> (IMHO).
> I exit on the right via let's say VXLAN. If metadata is naturally welded to
> encapsulation as you propose this means losing all of it.
> >
> >>> Geneve options are not specific to service chaining, though a service
> chain blob can be one of the option in Geneve header. So it does belongs to
> network virtualization encapsulation format and not to SFC charter.
> >> Disagree.
> >>
> >> 2. No cookies so no defence against misconfiguration 3. Unknown
> security interactions and no analysis of interactions with IPSEC so no
> clarity on extending beyond datacenter.
> common implementation feature allowing for same "metadata" blob behaviour.
> Specifying that blob to hit an aligned address so you can configure offload
> for it is trivial. It is not worth it to waste a draft (and a potential
> RFC) on what is an implementation detail. What if I implement an offload
> engine which works on unaligned addresses? Go and change the draft just
> because of that?
> >> So if it is "just an encapsulation" there is no reason for it to exist
> - it has all been done, we should just standardize the existing offset
> the key here is the way this is treated in a system - the meaning of life,
> universe and the metadata blob. That at the moment is being looked at only
> in SFC, so it does indeed belong there despite the fact that they are
> looking at a reduced set of meanings for that.
> >>
> >> A.
> >>
> >>
> >>
> >>> That work (chaining and metadata) is already being addressed in the
> SFC workgroup by charter.  At the very least, the draft should move if
> that's it's primary thrust.  In that work, there is no coupling between the
> header for forwarding in an overlay and the header for chaining and passing
> metadata Šand I think that was on purpose.
> >>>

---

From: Tom Herbert <[email-masked]>
Date: Fri, 28 Feb 2014 17:51:32 -0800

On Fri, Feb 28, 2014 at 3:24 PM, Sam Aldrin <[email-masked]> wrote:
> indicates it as informational. Also it is good to have it as draft-nvo3....,
> if it is meant for NVo3 WG.
> - I fail to find good reasoning, in the current version of the draft, on why
> design of encap transport header should be closely associated with metadata
> OR closely tied together? Could you add more details to clarify?

The draft alludes to the general need for extensibility, but does not
provide any example uses, so maybe I can suggest one. We have a real
use case for an encapsulation protocol with security to allow
validation of the virtual network identifier. In their current for
vxlan and nvgre have no provisions for authenticating or integrity
check of vni, existing mechanisms in the network were not deemed
robust enough to guarantee integrity of vni and ensure strict
isolation between tenants. UDP checksum is not sufficient for this. We
need a mechanism to at least have enforce an unpredictable security
token, or possibly at stronger authentication using something like a
message digest. This is intrinsic to the encapsulation, we cannot
deploy network virtualization without this security, hence an
extensible protocol is desirable. Additionally, as the network scales,
new threats emerge, we may have need for further extensions to adapt.
All of this needs to be efficient and amenable to HW performance
optimizations.

> - In the past WG sessions and over the emails, the indication given was that
> various encap technologies already exist. Defining a good control plane
> technology is the need for the WG, no?
> On Fri, Feb 28, 2014 at 11:00 AM, Anton Ivanov (antivano)
>> >> for it is trivial. It is not worth it to waste a draft (and a potential RFC)
>> >> on what is an implementation detail. What if I implement an offload engine
>> >> which works on unaligned addresses? Go and change the draft just because of
>> >> that?

---

From: "Anton Ivanov (antivano)" <[email-masked]>
Date: Sat, 1 Mar 2014 07:01:15 +0000

> All of this needs to be efficient and amenable to HW performance

Tom, you are describing the L2TPv3 cookie.
http://tools.ietf.org/html/rfc3931#section-4.1.1 That has already been
defined and standardized in 2005.

As quite a few people said - we do not need to invent a new
encapsulation for the goals of this draft or for the goals of NVO3 for
that matter. This just proves the point.

We can copy that option to VXLAN or NVGRE as an extension if we wish too.

As far as metadata extensions - I believe there is an agreement that we
should do it. Similarly there is a consensus that they should not be
welded into the network header. That particular aspect of the design has
no other function but to be a "mono-culture monopoly license".

A.