Subject: Re: [nvo3] RFC 7637 on NVGRE: Network Virtualization Using Generic Routing Encapsulation

From: Tom Herbert <[email-masked]>
Date: Thu, 1 Oct 2015 12:37:57 -0700

Hi Pankaj,

Do you think there is any value, intent, or issue for doing NVGRE/UDP
(via https://tools.ietf.org/html/draft-ietf-tsvwg-gre-in-udp-encap-07)

Tom

On Thu, Sep 24, 2015 at 1:43 PM, Pankaj Garg <[email-masked]> wrote:
> FYI, NVGRE is published as an information RFC 7637. Your documents that reference NVGRE, please use this RFC number.
>
> Thanks
> Pankaj
>
>> To: ietf-announce at ietf.org, rfc-dist at rfc-editor.org
>> Subject: RFC 7637 on NVGRE: Network Virtualization Using Generic Routing
>> Encapsulation
>> From: rfc-editor at rfc-editor.org
>> Date: Wed, 23 Sep 2015 15:16:09 -0700 (PDT)
>> Archived-at: <http://mailarchive.ietf.org/arch/msg/ietf-
>> announce/KPKrdzVjAGl5H931oM-D1ce71zQ>
>> Cc: rfc-editor at rfc-editor.org
>> Delivered-to: ietf-announce at ietfa.amsl.com
>> List-archive: <https://mailarchive.ietf.org/arch/browse/ietf-announce/>
>> List-help: <mailto:[email-masked]?subject=help>
>> List-id: "IETF announcement list. No discussions." <ietf-announce.ietf.org>
>> List-post: <mailto:[email-masked]>
>> List-subscribe: <https://www.ietf.org/mailman/listinfo/ietf-announce>,
>> <mailto:[email-masked]?subject=subscribe>
>> List-unsubscribe: <https://www.ietf.org/mailman/options/ietf-announce>,
>> <mailto:[email-masked]?subject=unsubscribe>
>> Reply-to: ietf at ietf.org
>>
>>   A new Request for Comments is now available in online RFC libraries.
>>
>>
>>         RFC 7637
>>
>>         Title:      NVGRE: Network Virtualization Using Generic
>>                     Routing Encapsulation
>>         Author:     P. Garg, Ed.,
>>                     Y. Wang, Ed.
>>         Status:     Informational
>>         Stream:     Independent
>>         Date:       September 2015
>>         Mailbox:    pankajg at microsoft.com,
>>                     yushwang at microsoft.com
>>         Pages:      17
>>         Characters: 40042
>>         Updates/Obsoletes/SeeAlso:   None
>>
>>         I-D Tag:    draft-sridharan-virtualization-nvgre-08.txt
>>
>>         URL:        https://www.rfc-editor.org/info/rfc7637
>>
>>         DOI:        http://dx.doi.org/10.17487/RFC7637
>>
>> This document describes the usage of the Generic Routing Encapsulation
>> (GRE) header for Network Virtualization (NVGRE) in multi-tenant data
>> centers.  Network Virtualization decouples virtual networks and addresses
>> from physical network infrastructure, providing isolation and concurrency
>> between multiple virtual networks on the same physical network
>> infrastructure.  This document also introduces a Network Virtualization
>> framework to illustrate the use cases, but the focus is on specifying the data-
>> plane aspect of NVGRE.
>>
>>
>> INFORMATIONAL: This memo provides information for the Internet
>> community.
>> It does not specify an Internet standard of any kind. Distribution of this
>> memo is unlimited.
>>
>> This announcement is sent to the IETF-Announce and rfc-dist lists.
>> To subscribe or unsubscribe, see
>>   https://www.ietf.org/mailman/listinfo/ietf-announce
>>   https://mailman.rfc-editor.org/mailman/listinfo/rfc-dist
>>
>> For searching the RFC series, see https://www.rfc-editor.org/search For
>> downloading RFCs, see https://www.rfc-editor.org/rfc.html
>>
>> Requests for special distribution should be addressed to either the author of
>> the RFC in question, or to rfc-editor at rfc-editor.org.  Unless specifically
>> noted otherwise on the RFC itself, all RFCs are for unlimited distribution.
>>
>>
>> The RFC Editor Team
>> Association Management Solutions, LLC
>
> _______________________________________________
> nvo3 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nvo3

---

From: Pankaj Garg <[email-masked]>
Date: Fri, 2 Oct 2015 20:31:05 +0000

Hi Tom,

In my personal opinion, for network virtualization, I don't see much value in GRE in UDP. Each new encapsulation takes a lot of effort and time before it achieves ecosystem support (from software to hardware to offloads) to make it a viable option. We already have VXLAN/GPE so I am not sure what we gain with GRE in UDP over VXLAN/VXLAN-GPE. There may be scenarios outside of network virtualization, where this may be useful, but I am not familiar with those.

Thanks
Pankaj

> -----Original Message-----
> From: Tom Herbert [mailto:[email-masked]]
> Sent: Thursday, October 1, 2015 12:38 PM
> To: Pankaj Garg <[email-masked]>
> Cc: [email-masked]
> Subject: Re: [nvo3] RFC 7637 on NVGRE: Network Virtualization Using
> Generic Routing Encapsulation
> 
> Hi Pankaj,
> 
> Do you think there is any value, intent, or issue for doing NVGRE/UDP (via
> https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2ftools.ie
> tf.org%2fhtml%2fdraft-ietf-tsvwg-gre-in-udp-encap-
> 07&data=01%7c01%7cpankajg%40microsoft.com%7cb4e8bc27b4104053ac7d0
> 8d2ca97cf9d%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=mjLRy8EUy
> dHFlg4CVx6wqAc0kuElYDl45P%2bNoPH2QaM%3d)
> 
> Tom
> 
> > FYI, NVGRE is published as an information RFC 7637. Your documents that
> reference NVGRE, please use this RFC number.
> >
> > Thanks
> > Pankaj
> >
> >> To: ietf-announce at
> >> https://na01.safelinks.protection.outlook.com/?url=ietf.org&data=01%7
> >>
> c01%7cpankajg%40microsoft.com%7cb4e8bc27b4104053ac7d08d2ca97cf9d%
> 7c72
> >>
> f988bf86f141af91ab2d7cd011db47%7c1&sdata=5hAuGhDIDQJKVz73tx1%2b7
> sWRI3
> >> KbhTGidy0PMcC2SHc%3d, rfc-dist at
> >> https://na01.safelinks.protection.outlook.com/?url=rfc-editor.org&dat
> >>
> a=01%7c01%7cpankajg%40microsoft.com%7cb4e8bc27b4104053ac7d08d2ca9
> 7cf9
> >>
> d%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=2cRTxuyq9Uzy4cM34
> %2fIi
> >> jExLAgN8PbJp4WbdxMAk%2bNU%3d
> >> Subject: RFC 7637 on NVGRE: Network Virtualization Using Generic
> >> Routing Encapsulation
> >> From: rfc-editor at
> >> https://na01.safelinks.protection.outlook.com/?url=rfc-editor.org&dat
> >>
> a=01%7c01%7cpankajg%40microsoft.com%7cb4e8bc27b4104053ac7d08d2ca9
> 7cf9
> >>
> d%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=2cRTxuyq9Uzy4cM34
> %2fIi
> >> jExLAgN8PbJp4WbdxMAk%2bNU%3d
> >> Archived-at:
> >> <https://na01.safelinks.protection.outlook.com/?url=http%3a%2f%2fmail
> >> archive.ietf.org%2farch%2fmsg%2fietf-
> &data=01%7c01%7cpankajg%40micros
> >>
> oft.com%7cb4e8bc27b4104053ac7d08d2ca97cf9d%7c72f988bf86f141af91ab2d
> 7c
> >>
> d011db47%7c1&sdata=awTaXvxdIhjCQwRST26DMgCILeQsfnypU%2bMcS%2
> bSn5QI%3d
> >> Cc: rfc-editor at
> >> https://na01.safelinks.protection.outlook.com/?url=rfc-editor.org&dat
> >>
> a=01%7c01%7cpankajg%40microsoft.com%7cb4e8bc27b4104053ac7d08d2ca9
> 7cf9
> >>
> d%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=2cRTxuyq9Uzy4cM34
> %2fIi
> >> jExLAgN8PbJp4WbdxMAk%2bNU%3d
> >> Delivered-to: ietf-announce at
> >> https://na01.safelinks.protection.outlook.com/?url=ietfa.amsl.com&dat
> >>
> a=01%7c01%7cpankajg%40microsoft.com%7cb4e8bc27b4104053ac7d08d2ca9
> 7cf9
> >>
> d%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=M%2fe6ro1epcVueeV
> 33Vce
> >> gEClYlei0kGxPJJYjkyJgQQ%3d
> >> List-archive:
> >>
> <https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fmai
> >> larchive.ietf.org%2farch%2fbrowse%2fietf-
> announce%2f&data=01%7c01%7cp
> >>
> ankajg%40microsoft.com%7cb4e8bc27b4104053ac7d08d2ca97cf9d%7c72f988
> bf8
> >>
> 6f141af91ab2d7cd011db47%7c1&sdata=ewINwsulb7pgkBwQEfLPn2C%2fG8V
> UJARLm
> >> 0tUxt9hDE0%3d>
> >> List-id: "IETF announcement list. No discussions."
> >> <https://na01.safelinks.protection.outlook.com/?url=ietf-announce.iet
> >>
> f.org&data=01%7c01%7cpankajg%40microsoft.com%7cb4e8bc27b4104053ac7
> d08
> >>
> d2ca97cf9d%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=1a9JG3L4tg
> brZ
> >> %2ff0QtyzPpamed6mC2jjDcg%2fKZe0To0%3d>
> >> List-subscribe:
> <https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fwww
> >> .ietf.org%2fmailman%2flistinfo%2fietf-
> announce&data=01%7c01%7cpankajg
> >>
> %40microsoft.com%7cb4e8bc27b4104053ac7d08d2ca97cf9d%7c72f988bf86f1
> 41a
> >>
> f91ab2d7cd011db47%7c1&sdata=lpYAfNFCgD0wGuCjHWjgvf9iJkv2Dx0EywPx
> 6ZJL3
> >> 7g%3d>, <mailto:[email-masked]?subject=subscribe>
> >> List-unsubscribe:
> >>
> <https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fwww
> >> .ietf.org%2fmailman%2foptions%2fietf-
> announce&data=01%7c01%7cpankajg%
> >>
> 40microsoft.com%7cb4e8bc27b4104053ac7d08d2ca97cf9d%7c72f988bf86f141
> af
> >>
> 91ab2d7cd011db47%7c1&sdata=mivKM9832k26InTio8KQsMLs3RPk4WX%2bN
> 6hk3mbm
> >> qbg%3d>, <mailto:ietf-announce-
> [email-masked]?subject=unsubscribe>
> >> Reply-to: ietf at
> >> https://na01.safelinks.protection.outlook.com/?url=ietf.org&data=01%7
> >>
> c01%7cpankajg%40microsoft.com%7cb4e8bc27b4104053ac7d08d2ca97cf9d%
> 7c72
> >>
> f988bf86f141af91ab2d7cd011db47%7c1&sdata=5hAuGhDIDQJKVz73tx1%2b7
> sWRI3
> >> KbhTGidy0PMcC2SHc%3d
> >>
> >>         Mailbox:    pankajg at
> https://na01.safelinks.protection.outlook.com/?url=microsoft.com&data=01
> %7c01%7cpankajg%40microsoft.com%7cb4e8bc27b4104053ac7d08d2ca97cf9
> d%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=Zv8JyiVSIMZ5jnuPJFa
> 7Km8aQuyEl5l3%2f2oZ09GkrJo%3d,
> >>                     yushwang at
> https://na01.safelinks.protection.outlook.com/?url=microsoft.com&data=01
> %7c01%7cpankajg%40microsoft.com%7cb4e8bc27b4104053ac7d08d2ca97cf9
> d%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=Zv8JyiVSIMZ5jnuPJFa
> 7Km8aQuyEl5l3%2f2oZ09GkrJo%3d
> >>         Pages:      17
> >>         Characters: 40042
> >>         URL:
> https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fwww.rf
> c-
> editor.org%2finfo%2frfc7637&data=01%7c01%7cpankajg%40microsoft.com%
> 7cb4e8bc27b4104053ac7d08d2ca97cf9d%7c72f988bf86f141af91ab2d7cd011db
> 47%7c1&sdata=1gLcmNAgGpxjG%2fSQGunY3F6SM4UJR7RdnOdnof3zLLo%3
> d
> >>
> >>         DOI:
> https://na01.safelinks.protection.outlook.com/?url=http%3a%2f%2fdx.doi.o
> rg%2f10.17487%2fRFC7637&data=01%7c01%7cpankajg%40microsoft.com%7c
> b4e8bc27b4104053ac7d08d2ca97cf9d%7c72f988bf86f141af91ab2d7cd011db47
> %7c1&sdata=ifenB7Oyzkra50G0JmwbU5aGPadkn877%2fCv3qSDxR2w%3d
> >>
> >> centers.  Network Virtualization decouples virtual networks and
> >> addresses from physical network infrastructure, providing isolation
> >> and concurrency between multiple virtual networks on the same
> >> physical network infrastructure.  This document also introduces a
> >> Network Virtualization framework to illustrate the use cases, but the
> >> focus is on specifying the data- plane aspect of NVGRE.
> >>
> >>
> >> It does not specify an Internet standard of any kind. Distribution of
> >> this memo is unlimited.
> >>
> https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fwww.i
> etf.org%2fmailman%2flistinfo%2fietf-
> announce&data=01%7c01%7cpankajg%40microsoft.com%7cb4e8bc27b41040
> 53ac7d08d2ca97cf9d%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=lpY
> AfNFCgD0wGuCjHWjgvf9iJkv2Dx0EywPx6ZJL37g%3d
> >>
> >> https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fmail
> >> man.rfc-editor.org%2fmailman%2flistinfo%2frfc-
> dist&data=01%7c01%7cpan
> >>
> kajg%40microsoft.com%7cb4e8bc27b4104053ac7d08d2ca97cf9d%7c72f988bf
> 86f
> >>
> 141af91ab2d7cd011db47%7c1&sdata=sUFsfHCWD56LRRX3uHF%2fn4%2f8LV5
> BBCr1M
> >> qlLcxpAOEA%3d
> >>
> >> For searching the RFC series, see
> >>
> https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fwww.
> >> rfc-
> editor.org%2fsearch&data=01%7c01%7cpankajg%40microsoft.com%7cb4e8
> >>
> bc27b4104053ac7d08d2ca97cf9d%7c72f988bf86f141af91ab2d7cd011db47%7c
> 1&s
> >> data=Dzfgq7%2fPmCe3QZQkLVAhawwqaC1%2flYiVFLNGdouOWr0%3d
> For
> >> downloading RFCs, see
> >>
> https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fwww.
> >> rfc-
> editor.org%2frfc.html&data=01%7c01%7cpankajg%40microsoft.com%7cb4
> >>
> e8bc27b4104053ac7d08d2ca97cf9d%7c72f988bf86f141af91ab2d7cd011db47%
> 7c1
> >> &sdata=yMHIW0ciMjjkpD5iwuIVYEIOE5%2bhOtchaHuyQiUJ9TU%3d
> >>
> >> Requests for special distribution should be addressed to either the
> >> author of the RFC in question, or to rfc-editor at rfc-editor.org.
> >> Unless specifically noted otherwise on the RFC itself, all RFCs are for
> unlimited distribution.
> >>
> >>
> >> The RFC Editor Team
> https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fwww.i
> >
> etf.org%2fmailman%2flistinfo%2fnvo3&data=01%7c01%7cpankajg%40micros
> oft
> >
> .com%7cb4e8bc27b4104053ac7d08d2ca97cf9d%7c72f988bf86f141af91ab2d7c
> d011
> >
> db47%7c1&sdata=XquLeptMFXZ4doDoOE8cFJuIuPR5zWDWXK49awfESBg%3
> d

---

From: "Manish Kumar (manishkr)" <[email-masked]>
Date: Tue, 6 Oct 2015 06:01:47 +0000

Hi Tom, Pankaj,

The usage of the UDP header in such encapsulations is mostly for entropy
(load-balancing), aid NAT traversal, etc. This indirectly makes the
transport believe that the packet is a regular IP packet (with L4 header
following the IP header) rather than a tunnelled IP packet(some
systems/functionalities always assumed this). In this context, it makes
sense to stamp he UDP header only on the outer most encap, the ones added
on the inner encaps don¹t add much value. Hence, I¹m of the view that the
UDP header be dissociated with individual encaps and just stamped after
the outer most encap when needed - the destination UDP port, as usual,
identifies the encap protocol. Given that GRE is, by far, the most
commonly used general purpose encap deployed today, it may be good to have
GRE in UDP. I would like to see this as GRE in UDP rather than GRE and
GRE+UDP as two different encaps conceptually (implementations may choose
to interpret based on what¹s convenient). Optional UDP would also mean,
those 8 bytes are only used when needed (which could be, for example,
communicated by the control plane if there is one).

Just my two cents!

Cheers,
Manish

On 03/10/15 2:01 am, "nvo3 on behalf of Pankaj Garg"
<[email-masked] on behalf of [email-masked]> wrote:

>Hi Tom,
>
>In my personal opinion, for network virtualization, I don't see much
>value in GRE in UDP. Each new encapsulation takes a lot of effort and
>time before it achieves ecosystem support (from software to hardware to
>offloads) to make it a viable option. We already have VXLAN/GPE so I am
>not sure what we gain with GRE in UDP over VXLAN/VXLAN-GPE. There may be
>scenarios outside of network virtualization, where this may be useful,
>but I am not familiar with those.
>
>Thanks
>Pankaj
>
>>(via

---

From: Lucy yong <[email-masked]>
Date: Tue, 6 Oct 2015 16:31:23 +0000

Hi Manish,

I agree with you. 

GRE/UDP encapsulation protocol can apply to the Internet and a well-managed operator network. NVGRE is adapted from GRE and made specific for Ethernet/IP payload. We should not make them as two distinct encap. protocols for this space. NVGRE can be benefit from GRE/UDP for flow entropy and UDP checksum in IPv6; thus should be considered as optional to use.

Lucy

-----Original Message-----
From: nvo3 [mailto:[email-masked]] On Behalf Of Manish Kumar (manishkr)
Sent: Tuesday, October 06, 2015 1:02 AM
To: Pankaj Garg; Tom Herbert
Cc: [email-masked]
Subject: Re: [nvo3] RFC 7637 on NVGRE: Network Virtualization Using Generic Routing Encapsulation

Hi Tom, Pankaj,

The usage of the UDP header in such encapsulations is mostly for entropy (load-balancing), aid NAT traversal, etc. This indirectly makes the transport believe that the packet is a regular IP packet (with L4 header following the IP header) rather than a tunnelled IP packet(some systems/functionalities always assumed this). In this context, it makes sense to stamp he UDP header only on the outer most encap, the ones added on the inner encaps don¹t add much value. Hence, I¹m of the view that the UDP header be dissociated with individual encaps and just stamped after the outer most encap when needed - the destination UDP port, as usual, identifies the encap protocol. Given that GRE is, by far, the most commonly used general purpose encap deployed today, it may be good to have GRE in UDP. I would like to see this as GRE in UDP rather than GRE and
GRE+UDP as two different encaps conceptually (implementations may choose
to interpret based on what¹s convenient). Optional UDP would also mean, those 8 bytes are only used when needed (which could be, for example, communicated by the control plane if there is one).

Just my two cents!

Cheers,
Manish

On 03/10/15 2:01 am, "nvo3 on behalf of Pankaj Garg"
<[email-masked] on behalf of [email-masked]> wrote:

>not sure what we gain with GRE in UDP over VXLAN/VXLAN-GPE. There may 
>be scenarios outside of network virtualization, where this may be 
>useful, but I am not familiar with those.
>
>Thanks
>Pankaj
>
>> rf
>> https://na01.safelinks.protection.outlook.com/?url=http%3a%2f%2fdx.do
>> i.o rg%2f10.17487%2fRFC7637&data=01%7c01%7cpankajg%40microsoft.com%7c
>> >> (GRE) header for Network Virtualization (NVGRE) in multi-tenant 
>> >> data centers.  Network Virtualization decouples virtual networks 
>> >> and addresses from physical network infrastructure, providing 
>> >> isolation and concurrency between multiple virtual networks on the 
>> >> same physical network infrastructure.  This document also 
>> >> introduces a Network Virtualization framework to illustrate the 
>> >> use cases, but the focus is on specifying the data- plane aspect of NVGRE.
>> >>
>> >>
>> >> It does not specify an Internet standard of any kind. Distribution 
>> >> of this memo is unlimited.
>> >>
>> i
>> >> Requests for special distribution should be addressed to either 
>> >> the author of the RFC in question, or to rfc-editor at rfc-editor.org.
>> >> Unless specifically noted otherwise on the RFC itself, all RFCs 
>> >> are
>>for
>> unlimited distribution.
>> >>
>> >>
>> >> The RFC Editor Team
>> i

---

From: Tom Herbert <[email-masked]>
Date: Tue, 6 Oct 2015 09:45:32 -0700

> GRE/UDP encapsulation protocol can apply to the Internet and a well-managed operator network. NVGRE is adapted from GRE and made specific for Ethernet/IP payload. We should not make them as two dis…
I you might still need to define NVGRE/UDP separately. In particular,
I wonder if NVGRE/UDP necessitate port number? AFAIK there is no way
to distinguish NVGRE from other uses of GRE, a different UDP port
number could allow that.

Tom

---

From: Lucy yong <[email-masked]>
Date: Tue, 6 Oct 2015 17:04:42 +0000

Tom,

Protocol Type in GRE header is used to determine NVGRE encap.. No need additional UDP port for NVGRE encap.

Lucy

-----Original Message-----
From: Tom Herbert [mailto:[email-masked]] 
Sent: Tuesday, October 06, 2015 11:46 AM
To: Lucy yong
Cc: Manish Kumar (manishkr); Pankaj Garg; [email-masked]
Subject: Re: [nvo3] RFC 7637 on NVGRE: Network Virtualization Using Generic Routing Encapsulation

> GRE/UDP encapsulation protocol can apply to the Internet and a well-managed operator network. NVGRE is adapted from GRE and made specific for Ethernet/IP payload. We should not make them as two dis…
I you might still need to define NVGRE/UDP separately. In particular, I wonder if NVGRE/UDP necessitate port number? AFAIK there is no way to distinguish NVGRE from other uses of GRE, a different UDP port number could allow that.

Tom

> The usage of the UDP header in such encapsulations is mostly for 
> entropy (load-balancing), aid NAT traversal, etc. This indirectly 
> makes the transport believe that the packet is a regular IP packet 
> (with L4 header following the IP header) rather than a tunnelled IP 
> packet(some systems/functionalities always assumed this). In this 
> context, it makes sense to stamp he UDP header only on the outer most 
> encap, the ones added on the inner encaps don¹t add much value. Hence, 
> I¹m of the view that the UDP header be dissociated with individual 
> encaps and just stamped after the outer most encap when needed - the 
> destination UDP port, as usual, identifies the encap protocol. Given 
> that GRE is, by far, the most commonly used general purpose encap 
> deployed today, it may be good to have GRE in UDP. I would like to see 
> this as GRE in UDP rather than GRE and
> GRE+UDP as two different encaps conceptually (implementations may 
> GRE+choose
>>offloads) to make it a viable option. We already have VXLAN/GPE so I 
>>am not sure what we gain with GRE in UDP over VXLAN/VXLAN-GPE. There 
>>may be scenarios outside of network virtualization, where this may be 
>>>https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2ftool
>>>s
>>>.ie
>>> >> https://na01.safelinks.protection.outlook.com/?url=ietf.org&data=
>>> >> 0
>>> >> 1%7
>>> >>
>>> >> https://na01.safelinks.protection.outlook.com/?url=rfc-editor.org
>>> >> &
>>> >> dat
>>> >>
>>> >> https://na01.safelinks.protection.outlook.com/?url=rfc-editor.org
>>> >> &
>>> >> dat
>>> >>
>>> >> <https://na01.safelinks.protection.outlook.com/?url=http%3a%2f%2f
>>> >> m
>>> >> ail
>>> >> https://na01.safelinks.protection.outlook.com/?url=rfc-editor.org
>>> >> &
>>> >> dat
>>> >>
>>> >> https://na01.safelinks.protection.outlook.com/?url=ietfa.amsl.com
>>> >> &
>>> >> dat
>>> >>
>>> >> https://na01.safelinks.protection.outlook.com/?url=ietf.org&data=
>>> >> 0
>>> >> 1%7
>>> >>
>>> https://na01.safelinks.protection.outlook.com/?url=microsoft.com&dat
>>> a
>>> =01
>>> https://na01.safelinks.protection.outlook.com/?url=microsoft.com&dat
>>> a
>>> =01
>>> https://na01.safelinks.protection.outlook.com/?url=http%3a%2f%2fdx.d
>>> o i.o 
>>> >> isolation and concurrency between multiple virtual networks on 
>>> >> the same physical network infrastructure.  This document also 
>>> >> It does not specify an Internet standard of any kind. 
>>> >> Distribution of this memo is unlimited.
>>> >>
>>> >> https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2f
>>> >> m
>>> >> ail

---

From: "Manish Kumar (manishkr)" <[email-masked]>
Date: Wed, 28 Oct 2015 08:42:24 +0000

Hi Tom,

Any specific reason that we want to distinguish NVGRE from other usages of
GRE (except for the usage of last 8 bits for entropy). In fact, given that
the meaning of key was relatively open, many implementations have
optionally used the last 8 bits of the key for entropy even before NVGRE
(DC usage) came into picture (with both ends of the tunnel in agreement).
If needed, we could use one of the reserved bits at the beginning of the
header to indicate such usage.

Alternatively, GRE over UDP gives all that we need. To be honest, I still
fail to find a reason of why we really needed VxLAN and VxLAN GPE(now)
given that GRE has already existed for so long (as well as some
implementations of GRE over UDP) - this is new learning (in terms of new
encap, new numbers, etc, etc!) and a lot of extra work for implementors
and users alike! Just carrying GRE over UDP (could be thought of as a
nested encap) along with using a few reserved bits (for OAM, etc) seems to
me to give all that we’ll get after a long arduous journey to VxLAN GPE
through VxLAN! May be my personal opinion but I haven’t seen anything
otherwise yet.

Thanks,
Manish

On 06/10/15 10:15 pm, "Tom Herbert" <[email-masked]> wrote:

>> GRE/UDP encapsulation protocol can apply to the Internet and a
>>well-managed operator network. NVGRE is adapted from GRE and made
>>specific for Ethernet/IP payload. We should not make them as two
>>distinct encap. protocols for this space. NVGRE can be benefit from
>>GRE/UDP for flow entropy and UDP checksum in IPv6; thus should be
>>considered as optional to use.
>>
>>entropy (load-balancing), aid NAT traversal, etc. This indirectly makes
>>the transport believe that the packet is a regular IP packet (with L4
>>header following the IP header) rather than a tunnelled IP packet(some
>>sense to stamp he UDP header only on the outer most encap, the ones
>>added on the inner encaps don¹t add much value. Hence, I¹m of the view
>>that the UDP header be dissociated with individual encaps and just
>>stamped after the outer most encap when needed - the destination UDP
>>port, as usual, identifies the encap protocol. Given that GRE is, by
>>far, the most commonly used general purpose encap deployed today, it may
>>be good to have GRE in UDP. I would like to see this as GRE in UDP
>>rather than GRE and

---

From: Tom Herbert <[email-masked]>
Date: Wed, 28 Oct 2015 08:55:35 -0700

> header to indicate such usage.
I don't see a deterministic way to distinguish a plain GRE tunnel that
has been configured from an NVGRE tunnel. As you point out there are
already implementations that use the keyid for other purposes.
NVGRE/UDP with a separate port number would clarify that.

> through VxLAN! May be my personal opinion but I haven’t seen anything
The showstopper for us using GRE for network virtualization is that it
is not extensible for adding new fields. In particular there is no way
(other than purposely overloading existing fields) to add a security
token to validate the VNID. This is why invented GUE which is more or
less just and extensible form of GRE.

Tom

---

From: Dino Farinacci <[email-masked]>
Date: Wed, 28 Oct 2015 10:10:52 -0700

You should ask the proponents of NVGRE, if they still have an investment in NVGRe. From what I hear, through the grapevine, those proponents use VXLAN to deliver services.

So with all due respect to the NVGRE advocates, would they object dropping NVGRE support from the nvo3 charter?

Practically speaking,
Dino

> On Oct 28, 2015, at 1:42 AM, Manish Kumar (manishkr) <[email-masked]> wrote:
> 
> Hi Tom,
>

---

From: Pankaj Garg <[email-masked]>
Date: Thu, 29 Oct 2015 19:10:27 +0000

NVGRE and VXLAN
NVGRE is widely used in datacenter networks and there is wide hardware support available for it (both in terms of NICs and Switches). The key advantage VXLAN has over NVGRE is that it allows larger entropy and doesn't require any change in middle boxes to calculate hash and do ECMP. Most network virtualization solutions are supporting both of these today. Even though in long term, things may settle on a single encap format, that is not practically possible in the short term. In my opinion, both NVGRE and VXLAN are here to stay for quite a while given the investment and usage of these technologies. Could we have settled on just one? Yes, we could have, but I am not sure either of these would be the one (read further).

Shortcomings:
Around maybe 2-3 years back, we identified that both of these encap have one severe limitation (well VXLAN had two) i.e. ability (or lack thereof) to extend them. In addition, VXLAN even did not allow encap of non-ethernet inner packet. The main problem for network virtualization, however was extensibility. There are a lot of use cases where such extensibility can add significant advantage to encapsulation. It seems other people were thinking of the same at the same time. This resulted in 3 encap options for "future" encap. VXLAN-GPE + NSH, Geneve and GUE.

VXLAN-GPE + NSH, Geneve and GUE
Out of the three, I personally prefer the first two, partly because I have been deeply involved in the extensibility design of these but more importantly I feel they offer advantage over GUE. VXLAN-GPE + NSH seems to be the most flexible of the present options as it allows extensibility in hardware and software friendly manner. When we think about extensibility, there is always struggle between hardware and software. While software can evolve much faster, hardware cannot and we have to design something that does not restrict software to innovate but eventually let hardware catch up and use those innovations as well.

A key limitation that prevents software from using extensions is NIC offloads. Both Geneve and VXLAN-GPE+NSH allows extension of these protocols without breaking NIC offloads. This is a huge advantage given that in a network virtualization environment, typically most endpoints are software. This allows software vendors to do faster innovation without worrying about the whole ecosystem aspects of it. Both Geneve and NSH provide this extensibility using TLV format and they both use exact same TLV format (by choice). 

However, not all endpoints can be software and eventually for some of those innovations (such as security or OAM etc.) it would be required (or useful) to have hardware participate. The key limitation of TLV format (that I have heard of) is that it is harder to parse in hardware at line rate. This is where I think NSH has an advantage. NSH allows us to define new MDType so that we can create fixed format headers (that are easier for hardware to parse than TLV). This allows faster innovation in software (using vendor specific TLV) and later on they can be converted into hardware friendly format using MDType.

GRE-in-UDP
I am not really sure where this fits in for network virtualization. It does not have required ecosystem support to be a viable option and does not solve the need for future encapsulations.

PS: I really hope that we (nvo3 group) picks up one option out of the three "future" encap as the standard option to avoid unnecessary fragmentation and engineering overhead.

Thanks
Pankaj 

> From: Dino Farinacci [mailto:[email-masked]]
> Sent: Wednesday, October 28, 2015 10:11 AM
> To: Manish Kumar (manishkr) <[email-masked]>
> Cc: Tom Herbert <[email-masked]>; Lucy Yong
> <[email-masked]>; Pankaj Garg <[email-masked]>;
> [email-masked]
> You should ask the proponents of NVGRE, if they still have an investment in
> NVGRe. From what I hear, through the grapevine, those proponents use
> VXLAN to deliver services.
> 
> > Any specific reason that we want to distinguish NVGRE from other
> > usages of GRE (except for the usage of last 8 bits for entropy). In
> > fact, given that the meaning of key was relatively open, many
> > implementations have optionally used the last 8 bits of the key for
> > entropy even before NVGRE (DC usage) came into picture (with both ends
> of the tunnel in agreement).
> > If needed, we could use one of the reserved bits at the beginning of
> > the header to indicate such usage.
> >
> > Alternatively, GRE over UDP gives all that we need. To be honest, I
> > still fail to find a reason of why we really needed VXLAN and VXLAN
> > GPE(now) given that GRE has already existed for so long (as well as
> > some implementations of GRE over UDP) - this is new learning (in terms
> > of new encap, new numbers, etc, etc!) and a lot of extra work for
> > implementors and users alike! Just carrying GRE over UDP (could be
> > thought of as a nested encap) along with using a few reserved bits
> > (for OAM, etc) seems to me to give all that we’ll get after a long
> > arduous journey to VXLAN GPE through VXLAN! May be my personal
> opinion
> > but I haven’t seen anything otherwise yet.
> >
> > Thanks,
> > Manish
> >
> >>> context, it makes sense to stamp he UDP header only on the outer
> >>> most encap, the ones added on the inner encaps don¹t add much value.
> >>> Hence, I¹m of the view that the UDP header be dissociated with
> >>> individual encaps and just stamped after the outer most encap when
> >>> needed - the destination UDP port, as usual, identifies the encap
> >>> protocol. Given that GRE is, by far, the most commonly used general
> >>> purpose encap deployed today, it may be good to have GRE in UDP. I
> >>> would like to see this as GRE in UDP rather than GRE and
> >>> to interpret based on what¹s convenient). Optional UDP would also
> >>> mean, those 8 bytes are only used when needed (which could be, for
> >>> example, communicated by the control plane if there is one).
> >>>
> >>> Just my two cents!
> >>>
> >>> Cheers,
> >>> Manish
> >>>
> >>>> In my personal opinion, for network virtualization, I don't see
> >>>> much value in GRE in UDP. Each new encapsulation takes a lot of
> >>>> effort and time before it achieves ecosystem support (from software
> >>>> to hardware to
> >>>> offloads) to make it a viable option. We already have VXLAN/GPE so
> >>>> I am not sure what we gain with GRE in UDP over VXLAN/VXLAN-GPE.
> >>>> There may be scenarios outside of network virtualization, where
> >>>> this may be useful, but I am not familiar with those.
> >>>>
> >>>> Thanks
> >>>> Pankaj
> >>>>
> https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2ft
> >>>>> ools
> >>>>> .ie
> >>>>> On Thu, Sep 24, 2015 at 1:43 PM, Pankaj Garg
> >>>>> <[email-masked]>
> >>>>> wrote:
> >>>>>> FYI, NVGRE is published as an information RFC 7637. Your
> >>>>>> documents
> >>>>> that
> >>>>>>> https://na01.safelinks.protection.outlook.com/?url=ietf.org&data
> >>>>>>> =0
> >>>>>>> 1%7
> >>>>>>>
> >>>>>
> >>>>>>> https://na01.safelinks.protection.outlook.com/?url=rfc-editor.or
> >>>>>>> g&
> >>>>>>> dat
> >>>>>>>
> >>>>>
> >>>>>>> https://na01.safelinks.protection.outlook.com/?url=rfc-editor.or
> >>>>>>> g&
> >>>>>>> dat
> >>>>>>>
> >>>>>
> <https://na01.safelinks.protection.outlook.com/?url=http%3a%2f%2
> >>>>>>> fm
> >>>>>>> ail
> >>>>>>> https://na01.safelinks.protection.outlook.com/?url=rfc-editor.or
> >>>>>>> g&
> >>>>>>> dat
> >>>>>>>
> >>>>>
> >>>>>>> https://na01.safelinks.protection.outlook.com/?url=ietfa.amsl.co
> >>>>>>> m&
> >>>>>>> dat
> >>>>>>>
> >>>>>
> >>>>>>> <https://na01.safelinks.protection.outlook.com/?url=ietf-
> announce.
> >>>>>>> iet
> >>>>>>>
> >>>>>
> >>>>>>> 7g%3d>,
> >>>>>>> https://na01.safelinks.protection.outlook.com/?url=ietf.org&data
> >>>>>>> =0
> >>>>>>> 1%7
> >>>>>>>
> >>>>>
> >>>>> https://na01.safelinks.protection.outlook.com/?url=microsoft.com&d
> >>>>> ata
> >>>>> =01
> >>>>>
> >>>>> https://na01.safelinks.protection.outlook.com/?url=microsoft.com&d
> >>>>> ata
> >>>>> =01
> >>>>>
> https://na01.safelinks.protection.outlook.com/?url=http%3a%2f%2fdx
> >>>>> .do i.o
> >>>>>
> 53ac7d08d2ca97cf9d%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=lp
> >>>>> Y AfNFCgD0wGuCjHWjgvf9iJkv2Dx0EywPx6ZJL37g%3d
> >>>>>>>
> >>>>>>>
> https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2
> >>>>>>> fm
> >>>>>>> ail
> https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fww
> >>>>
> w.ietf.org%2fmailman%2flistinfo%2fnvo3&data=01%7c01%7cpankajg%40mic
> >>>>
> rosoft.com%7cc7388153bdda4c0c55e808d2dfbab94b%7c72f988bf86f141af91a
> >>>>
> b2d7cd011db47%7c1&sdata=e9lIOaIf7f%2f0C4%2baNkbnjdXqptoSerbw%2fy
> Vxl
> >>>> Q%2bULD4%3d
> >>>
> https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fwww
> >>>
> .ietf.org%2fmailman%2flistinfo%2fnvo3&data=01%7c01%7cpankajg%40micro
> >>>
> soft.com%7cc7388153bdda4c0c55e808d2dfbab94b%7c72f988bf86f141af91ab
> 2d
> >>>
> 7cd011db47%7c1&sdata=e9lIOaIf7f%2f0C4%2baNkbnjdXqptoSerbw%2fyVxl
> Q%2b
> >>> ULD4%3d
> >
> .com%7cc7388153bdda4c0c55e808d2dfbab94b%7c72f988bf86f141af91ab2d7c
> d011
> >
> db47%7c1&sdata=e9lIOaIf7f%2f0C4%2baNkbnjdXqptoSerbw%2fyVxlQ%2bU
> LD4%3d

---

From: Tom Herbert <[email-masked]>
Date: Thu, 29 Oct 2015 12:24:31 -0700

> A key limitation that prevents software from using extensions is NIC offloads. Both Geneve and VXLAN-GPE+NSH allows extension of these protocols without breaking NIC offloads.

Can you describe why you think this is? Both Geneve and VXLAN-GPE+NSH
are not usable with most implementations of checksum offloads and
probably TSO. As we demonstrated in GUE it is possible to offload
encapsulated checksums by enabling the other UDP checksum and using
the remote checksum offload option (which we also implemented for
VXLAN).

Tom

---

From: Pankaj Garg <[email-masked]>
Date: Thu, 29 Oct 2015 19:41:03 +0000

Inline.

> Sent: Thursday, October 29, 2015 12:25 PM
> Cc: Dino Farinacci <[email-masked]>; Manish Kumar (manishkr)
> <[email-masked]>; Lucy Yong <[email-masked]>;
> [email-masked]
> > A key limitation that prevents software from using extensions is NIC
> offloads. Both Geneve and VXLAN-GPE+NSH allows extension of these
> protocols without breaking NIC offloads.
> 
> encapsulated checksums by enabling the other UDP checksum and using the
> remote checksum offload option (which we also implemented for VXLAN).
[PG] We can define newer TLVs and extend Geneve or NSH without breaking NIC offloads. NIC has to support Geneve/NSH offload in the first place, but after that we can safely extend using TLVs as needed without dealing with NIC offload changes etc. This includes the range of offload from Checksum, LSO, LRO, VMQ, RSS etc.

---

From: Tom Herbert <[email-masked]>
Date: Thu, 29 Oct 2015 12:55:07 -0700

> [PG] We can define newer TLVs and extend Geneve or NSH without breaking NIC offloads. NIC has to support Geneve/NSH offload in the first place, but after that we can safely extend using TLVs as nee…

Okay, you are assuming that we would go out an buy NICs that
specifically support these protocols in the first place. My belief is
that if NIC vendors implement generic offload mechanisms it is a far a
better route and could support a multitude of encapsulation protocols
(see http://people.netfilter.org/pablo/netdev0.1/papers/UDP-Encapsulation-in-Linux.pdf).

As for "safely extend using TLVs" have you actually verified that
works with HW, performance is unaffected, and this does not create new
vectors of DOS attacks? (Given the unmitigated disappointment with IP
options I'm very skeptical of and deployment of TLVs at L3 or below in
the data center.)

Thanks,
Tom

---

From: Pankaj Garg <[email-masked]>
Date: Thu, 29 Oct 2015 20:19:11 +0000

Inline.

> Sent: Thursday, October 29, 2015 12:55 PM
> > [PG] We can define newer TLVs and extend Geneve or NSH without
> breaking NIC offloads. NIC has to support Geneve/NSH offload in the first
> place, but after that we can safely extend using TLVs as needed without
> dealing with NIC offload changes etc. This includes the range of offload from
> Checksum, LSO, LRO, VMQ, RSS etc.
> 
> Okay, you are assuming that we would go out an buy NICs that specifically
> support these protocols in the first place. My belief is that if NIC vendors
> implement generic offload mechanisms it is a far a better route and could
> support a multitude of encapsulation protocols (see
[PG] It is up to hardware vendors to decide how to optimally and/or generically do this in hardware, but from software interface perspective, there is non-trivial amount of work that needs to be done for each new offload. It took significant time to get NVGRE and VXLAN offloads working correctly. We don't want to go through that pain, at least not unless we _have_ to and that is the promise of NSH and Geneve. If we can build a generic interface for offloads that NICs implement, that would be great as well, but I am skeptical of the viability of that given the variation in formats of various protocols.
> https://na01.safelinks.protection.outlook.com/?url=http%3a%2f%2fpeople.
> netfilter.org%2fpablo%2fnetdev0.1%2fpapers%2fUDP-Encapsulation-in-
> Linux.pdf&data=01%7c01%7cpankajg%40microsoft.com%7cc9ceeee9e5f6433
> 9c07c08d2e09ad904%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=MS
> vIxq6ldUFWP5v0g%2foPBZuC95iJYWPHfirUdYeA%2bPQ%3d).
[PG] I haven't read this fully yet, but will read it sometime.
> 
> As for "safely extend using TLVs" have you actually verified that works with
> HW, performance is unaffected, and this does not create new vectors of DOS
> attacks? (Given the unmitigated disappointment with IP options I'm very
> skeptical of and deployment of TLVs at L3 or below in the data center.)
[PG] We have not tested it but maybe some IHVs who are supporting Geneve (or NSH) offload can comment. I have no technical evidence, though, as to why it would _not_ work (sans some implementation constraints on maximum header size etc, which is pretty reasonable).

---

From: Lucy yong <[email-masked]>
Date: Thu, 29 Oct 2015 20:23:50 +0000

"GRE-in-UDP
I am not really sure where this fits in for network virtualization. It does not have required ecosystem support to be a viable option and does not solve the need for future encapsulations."

Counter argument is that there are many GRE applications today that face load balancing issue. GRE-in-UDP encap addresses this issue. NVGRE disadvantage mentioned below is addressed by GRE-in-UDP. In other words, NVGRE can benefit from the gre-in-udp if it will stay long.

Lucy    

-----Original Message-----
From: Pankaj Garg [mailto:[email-masked]] 
Sent: Thursday, October 29, 2015 2:10 PM
To: Dino Farinacci; Manish Kumar (manishkr)
Cc: Tom Herbert; Lucy yong; [email-masked]
Subject: RE: [nvo3] RFC 7637 on NVGRE: Network Virtualization Using Generic Routing Encapsulation

NVGRE and VXLAN

Shortcomings:

GRE-in-UDP

Thanks
Pankaj 

> You should ask the proponents of NVGRE, if they still have an 
> investment in NVGRe. From what I hear, through the grapevine, those 
> proponents use VXLAN to deliver services.
> 
> So with all due respect to the NVGRE advocates, would they object 
> dropping NVGRE support from the nvo3 charter?
> 
> Practically speaking,
> Dino
> 
> > some implementations of GRE over UDP) - this is new learning (in 
> > terms of new encap, new numbers, etc, etc!) and a lot of extra work 
> > for implementors and users alike! Just carrying GRE over UDP (could 
> > be thought of as a nested encap) along with using a few reserved 
> > bits (for OAM, etc) seems to me to give all that we’ll get after a 
> > long arduous journey to VXLAN GPE through VXLAN! May be my personal
> opinion
> >>> distinct encap. protocols for this space. NVGRE can be benefit 
> >>> from GRE/UDP for flow entropy and UDP checksum in IPv6; thus 
> >>> should be considered as optional to use.
> >>>
> >> I you might still need to define NVGRE/UDP separately. In 
> >> particular, I wonder if NVGRE/UDP necessitate port number? AFAIK 
> >> there is no way to distinguish NVGRE from other uses of GRE, a 
> >> different UDP port number could allow that.
> >>
> >> Tom
> >>
> >>> Lucy
> >>>
> >>> From: nvo3 [mailto:[email-masked]] On Behalf Of Manish 
> >>> Kumar
> >>> (manishkr)
> >>> (with L4 header following the IP header) rather than a tunnelled 
> >>> IP packet(some systems/functionalities always assumed this). In 
> >>> this context, it makes sense to stamp he UDP header only on the 
> >>> outer most encap, the ones added on the inner encaps don¹t add much value.
> >>> protocol. Given that GRE is, by far, the most commonly used 
> >>> general purpose encap deployed today, it may be good to have GRE 
> >>> in UDP. I would like to see this as GRE in UDP rather than GRE and
> >>>> effort and time before it achieves ecosystem support (from 
> >>>> software to hardware to
> >>>> offloads) to make it a viable option. We already have VXLAN/GPE 
> >>>> so I am not sure what we gain with GRE in UDP over VXLAN/VXLAN-GPE.
> >>>>>>> Subject: RFC 7637 on NVGRE: Network Virtualization Using 
> >>>>>>> List-help: 
> >>>>>>> <mailto:[email-masked]?subject=help>
> >>>>>>> (GRE) header for Network Virtualization (NVGRE) in 
> >>>>>>> multi-tenant data centers.  Network Virtualization decouples 
> >>>>>>> virtual networks and addresses from physical network 
> >>>>>>> infrastructure, providing isolation and concurrency between 
> >>>>>>> multiple virtual networks on the same physical network 
> >>>>>>> infrastructure.  This document also introduces a Network 
> >>>>>>> Virtualization framework to illustrate the use cases, but the 
> >>>>>>> Requests for special distribution should be addressed to 
> >>>>>>> either the author of the RFC in question, or to rfc-editor at
> >>>>> rfc-editor.org.
> >>>>>>> Unless specifically noted otherwise on the RFC itself, all 
> >>>>>>> RFCs are
> >>>>> for
> >>>>> unlimited distribution.
> >>>>>>>
> >>>>>>>
> >>>>>>> The RFC Editor Team

---

From: Tom Herbert <[email-masked]>
Date: Thu, 29 Oct 2015 13:34:17 -0700

> [PG] It is up to hardware vendors to decide how to optimally and/or generically do this in hardware, but from software interface perspective, there is non-trivial amount of work that needs to be do…

The generic interfaces already exists. Please look at my paper on UDP
encapsulation. We have simple well defined interfaces to handle four
of the five basic NIC offloads in a generic way (RX csum, TX csum,
RSS, and LSO). The fifth, LRO, would require NIC support per protocol
but it's pretty straightforward to make a software cognate for that
(but LRO has it own problems anyway...). This applies to VXLAN, GUE,
geneve, MPLS/UDP, GRE/UDP, IP/UDP and whatever new encapsulations we
would dream up.

Tom

---

From: Pankaj Garg <[email-masked]>
Date: Thu, 29 Oct 2015 20:38:34 +0000

Inline.

> Sent: Thursday, October 29, 2015 1:34 PM
> >> >> encapsulated checksums by enabling the other UDP checksum and
> >> >> using the remote checksum offload option (which we also
> >> >> implemented for
> >> VXLAN).
> >> breaking NIC offloads. NIC has to support Geneve/NSH offload in the
> >> first place, but after that we can safely extend using TLVs as needed
> >> without dealing with NIC offload changes etc. This includes the range
> >> of offload from Checksum, LSO, LRO, VMQ, RSS etc.
> >>
> >> that if NIC vendors implement generic offload mechanisms it is a far
> >> a better route and could support a multitude of encapsulation
> >> protocols (see
> > [PG] It is up to hardware vendors to decide how to optimally and/or
> generically do this in hardware, but from software interface perspective,
> there is non-trivial amount of work that needs to be done for each new
> offload. It took significant time to get NVGRE and VXLAN offloads working
> correctly. We don't want to go through that pain, at least not unless we
> _have_ to and that is the promise of NSH and Geneve. If we can build a
> generic interface for offloads that NICs implement, that would be great as
> well, but I am skeptical of the viability of that given the variation in formats of
> various protocols.
> 
> encapsulation. We have simple well defined interfaces to handle four of the
> five basic NIC offloads in a generic way (RX csum, TX csum, RSS, and LSO). The
> fifth, LRO, would require NIC support per protocol but it's pretty
> straightforward to make a software cognate for that (but LRO has it own
> problems anyway...). This applies to VXLAN, GUE, geneve, MPLS/UDP,
> GRE/UDP, IP/UDP and whatever new encapsulations we would dream up.

[PG] Yes I plan to read it, it looks pretty interesting paper, appreciate the link.

> >> works with HW, performance is unaffected, and this does not create
> >> new vectors of DOS attacks? (Given the unmitigated disappointment
> >> with IP options I'm very skeptical of and deployment of TLVs at L3 or
> >> below in the data center.)
> > [PG] We have not tested it but maybe some IHVs who are supporting
> Geneve (or NSH) offload can comment. I have no technical evidence,
> though, as to why it would _not_ work (sans some implementation
> constraints on maximum header size etc, which is pretty reasonable).
> >>
> >> Thanks,
> >> Tom
> >>
> >> >>
> >> >> Tom

---

From: Pankaj Garg <[email-masked]>
Date: Thu, 29 Oct 2015 20:41:15 +0000

> From: Lucy yong [mailto:[email-masked]]
> Sent: Thursday, October 29, 2015 1:24 PM
> To: Pankaj Garg <[email-masked]>; Dino Farinacci
> <[email-masked]>; Manish Kumar (manishkr) <[email-masked]>
> Cc: Tom Herbert <[email-masked]>; [email-masked]
> I am not really sure where this fits in for network virtualization. It does not
> have required ecosystem support to be a viable option and does not solve
> the need for future encapsulations."
> 
> Counter argument is that there are many GRE applications today that face
> load balancing issue. GRE-in-UDP encap addresses this issue. NVGRE
> disadvantage mentioned below is addressed by GRE-in-UDP. In other words,
> NVGRE can benefit from the gre-in-udp if it will stay long.

[PG] What is the advantage of moving from NVGRE to GRE-in-UDP, as opposed to moving to VXLAN? VXLAN has much better ecosystem support so one would immediately benefit from that, no? Outside of network virtualization GRE-in-UDP may have use cases and I have never questioned those as I am not familiar with those. 

> NVGRE is widely used in datacenter networks and there is wide hardware
> support available for it (both in terms of NICs and Switches). The key
> advantage VXLAN has over NVGRE is that it allows larger entropy and doesn't
> require any change in middle boxes to calculate hash and do ECMP. Most
> network virtualization solutions are supporting both of these today. Even
> though in long term, things may settle on a single encap format, that is not
> practically possible in the short term. In my opinion, both NVGRE and VXLAN
> are here to stay for quite a while given the investment and usage of these
> technologies. Could we have settled on just one? Yes, we could have, but I
> am not sure either of these would be the one (read further).
> 
> Shortcomings:
> Around maybe 2-3 years back, we identified that both of these encap have
> one severe limitation (well VXLAN had two) i.e. ability (or lack thereof) to
> extend them. In addition, VXLAN even did not allow encap of non-ethernet
> inner packet. The main problem for network virtualization, however was
> extensibility. There are a lot of use cases where such extensibility can add
> significant advantage to encapsulation. It seems other people were thinking
> of the same at the same time. This resulted in 3 encap options for "future"
> encap. VXLAN-GPE + NSH, Geneve and GUE.
> 
> Out of the three, I personally prefer the first two, partly because I have been
> deeply involved in the extensibility design of these but more importantly I
> feel they offer advantage over GUE. VXLAN-GPE + NSH seems to be the
> most flexible of the present options as it allows extensibility in hardware and
> software friendly manner. When we think about extensibility, there is always
> struggle between hardware and software. While software can evolve much
> faster, hardware cannot and we have to design something that does not
> restrict software to innovate but eventually let hardware catch up and use
> those innovations as well.
> 
> A key limitation that prevents software from using extensions is NIC offloads.
> Both Geneve and VXLAN-GPE+NSH allows extension of these protocols
> without breaking NIC offloads. This is a huge advantage given that in a
> network virtualization environment, typically most endpoints are software.
> This allows software vendors to do faster innovation without worrying about
> the whole ecosystem aspects of it. Both Geneve and NSH provide this
> extensibility using TLV format and they both use exact same TLV format (by
> choice).
> 
> However, not all endpoints can be software and eventually for some of
> those innovations (such as security or OAM etc.) it would be required (or
> useful) to have hardware participate. The key limitation of TLV format (that I
> have heard of) is that it is harder to parse in hardware at line rate. This is
> where I think NSH has an advantage. NSH allows us to define new MDType
> so that we can create fixed format headers (that are easier for hardware to
> parse than TLV). This allows faster innovation in software (using vendor
> specific TLV) and later on they can be converted into hardware friendly
> format using MDType.
> 
> GRE-in-UDP
> I am not really sure where this fits in for network virtualization. It does not
> have required ecosystem support to be a viable option and does not solve
> the need for future encapsulations.
> 
> PS: I really hope that we (nvo3 group) picks up one option out of the three
> "future" encap as the standard option to avoid unnecessary fragmentation
> and engineering overhead.
> 
> Thanks
> Pankaj
> 
> 
>

---

From: Lucy yong <[email-masked]>
Date: Fri, 30 Oct 2015 00:19:35 +0000

> 
> "GRE-in-UDP
> I am not really sure where this fits in for network virtualization. It 
> does not have required ecosystem support to be a viable option and 
> does not solve the need for future encapsulations."
> 
> Counter argument is that there are many GRE applications today that 
> face load balancing issue. GRE-in-UDP encap addresses this issue. 
> NVGRE disadvantage mentioned below is addressed by GRE-in-UDP. In 
> other words, NVGRE can benefit from the gre-in-udp if it will stay long.

[PG] What is the advantage of moving from NVGRE to GRE-in-UDP, as opposed to moving to VXLAN? VXLAN has much better ecosystem support so one would immediately benefit from that, no? Outside of network virtualization GRE-in-UDP may have use cases and I have never questioned those as I am not familiar with those. 
[Lucy]  VXLAN even did not allow encap of non-ethernet inner packet. -Lucy

> advantage VXLAN has over NVGRE is that it allows larger entropy and 
> doesn't require any change in middle boxes to calculate hash and do 
> ECMP. Most network virtualization solutions are supporting both of 
> these today. Even though in long term, things may settle on a single 
> encap format, that is not practically possible in the short term. In 
> my opinion, both NVGRE and VXLAN are here to stay for quite a while 
> given the investment and usage of these technologies. Could we have 
> settled on just one? Yes, we could have, but I am not sure either of these would be the one (read further).
> 
> Shortcomings:
> Around maybe 2-3 years back, we identified that both of these encap 
> have one severe limitation (well VXLAN had two) i.e. ability (or lack 
> thereof) to extend them. In addition, VXLAN even did not allow encap 
> of non-ethernet inner packet. The main problem for network 
> virtualization, however was extensibility. There are a lot of use 
> cases where such extensibility can add significant advantage to 
> encapsulation. It seems other people were thinking of the same at the same time. This resulted in 3 encap options for "future"
> Out of the three, I personally prefer the first two, partly because I 
> have been deeply involved in the extensibility design of these but 
> more importantly I feel they offer advantage over GUE. VXLAN-GPE + NSH 
> seems to be the most flexible of the present options as it allows 
> extensibility in hardware and software friendly manner. When we think 
> about extensibility, there is always struggle between hardware and 
> software. While software can evolve much faster, hardware cannot and 
> we have to design something that does not restrict software to 
> innovate but eventually let hardware catch up and use those innovations as well.
> 
> without breaking NIC offloads. This is a huge advantage given that in 
> a network virtualization environment, typically most endpoints are software.
> This allows software vendors to do faster innovation without worrying 
> about the whole ecosystem aspects of it. Both Geneve and NSH provide 
> this extensibility using TLV format and they both use exact same TLV 
> format (by choice).
> 
> useful) to have hardware participate. The key limitation of TLV format 
> (that I have heard of) is that it is harder to parse in hardware at 
> line rate. This is where I think NSH has an advantage. NSH allows us 
> to define new MDType so that we can create fixed format headers (that 
> are easier for hardware to parse than TLV). This allows faster 
> innovation in software (using vendor specific TLV) and later on they 
> can be converted into hardware friendly format using MDType.
> 
> GRE-in-UDP
> I am not really sure where this fits in for network virtualization. It 
> does not have required ecosystem support to be a viable option and 
> does not solve the need for future encapsulations.
> 
> PS: I really hope that we (nvo3 group) picks up one option out of the 
> three "future" encap as the standard option to avoid unnecessary 
> fragmentation and engineering overhead.
> 
> Thanks
> Pankaj
> 
> 
> 
> > > usages of GRE (except for the usage of last 8 bits for entropy). 
> > > In fact, given that the meaning of key was relatively open, many 
> > > implementations have optionally used the last 8 bits of the key 
> > > for entropy even before NVGRE (DC usage) came into picture (with 
> > > both ends
> > > If needed, we could use one of the reserved bits at the beginning 
> > > of the header to indicate such usage.
> > >
> > > Alternatively, GRE over UDP gives all that we need. To be honest, 
> > > I still fail to find a reason of why we really needed VXLAN and 
> > > VXLAN
> > > GPE(now) given that GRE has already existed for so long (as well 
> > > as some implementations of GRE over UDP) - this is new learning 
> > > (in terms of new encap, new numbers, etc, etc!) and a lot of extra 
> > > work for implementors and users alike! Just carrying GRE over UDP 
> > > (could be thought of as a nested encap) along with using a few 
> > > reserved bits (for OAM, etc) seems to me to give all that we’ll 
> > > get after a long arduous journey to VXLAN GPE through VXLAN! May 
> > > be my personal
> > opinion
> > >>> entropy (load-balancing), aid NAT traversal, etc. This 
> > >>> indirectly makes the transport believe that the packet is a 
> > >>> regular IP packet (with L4 header following the IP header) 
> > >>> rather than a tunnelled IP packet(some systems/functionalities 
> > >>> always assumed this). In this context, it makes sense to stamp 
> > >>> he UDP header only on the outer most encap, the ones added on 
> > >>> the inner encaps don¹t add
> much value.
> > >>> individual encaps and just stamped after the outer most encap 
> > >>> when needed - the destination UDP port, as usual, identifies the 
> > >>> encap protocol. Given that GRE is, by far, the most commonly 
> > >>> used general purpose encap deployed today, it may be good to 
> > >>> have GRE in UDP. I would like to see this as GRE in UDP rather 
> > >>> than GRE and
> > >>> GRE+UDP as two different encaps conceptually (implementations 
> > >>> GRE+may choose
> > >>> to interpret based on what¹s convenient). Optional UDP would 
> > >>> also mean, those 8 bytes are only used when needed (which could 
> > >>> be, for example, communicated by the control plane if there is one).
> > >>>
> > >>> Just my two cents!
> > >>>
> > >>> Cheers,
> > >>> Manish
> > >>>
> > >>>>>>> https://na01.safelinks.protection.outlook.com/?url=ietf.org&
> > >>>>>>> da
> > >>>>>>> ta
> > >>>>>>> =0
> > >>>>>>> 1%7
> > >>>>>>>
> > >>>>>
> >
> > >>>>>>> https://na01.safelinks.protection.outlook.com/?url=ietf.org&
> > >>>>>>> da
> > >>>>>>> ta
> > >>>>>>> =0
> > >>>>>>> 1%7
> > >>>>>>>
> > >>>>>
> >
> > >>>>> https://na01.safelinks.protection.outlook.com/?url=microsoft.c
> > >>>>> om
> > >>>>> &d
> > >>>>> ata
> > >>>>> =01
> > >>>>>
> >
> > >>>>> https://na01.safelinks.protection.outlook.com/?url=microsoft.c
> > >>>>> om
> > >>>>> &d
> > >>>>> ata
> > >>>>> =01
> > >>>>>
> >
> > >>>>>>> Virtualization framework to illustrate the use cases, but 
> > >>>>>>> the focus is on specifying the data- plane aspect
> > >>>>> of NVGRE.
> > >>>>>>>
> > >>>>>>>

---

From: Tom Herbert <[email-masked]>
Date: Thu, 29 Oct 2015 18:06:32 -0700

> [Lucy]  VXLAN even did not allow encap of non-ethernet inner packet. -Lucy

And GRE has been around for many years, has seen considerable
deployment and operational experience, and demonstrates a HW feasible
mechanism of extensibility for a low level protocol. In the end, both
NVGRE and VXLAN suffer from the same limitation in lack of
extensibility (VXLAN more than GRE), but NVGRE over UDP seems
reasonable to at least define if there is a deployed base of NVGRE
that could benefit.

Tom

---

From: Pankaj Garg <[email-masked]>
Date: Fri, 30 Oct 2015 05:13:29 +0000

Inline

> Sent: Thursday, October 29, 2015 6:07 PM
> To: Lucy yong <[email-masked]>
> Cc: Pankaj Garg <[email-masked]>; Dino Farinacci
> <[email-masked]>; Manish Kumar (manishkr) <[email-masked]>;
> [email-masked]
> >> I am not really sure where this fits in for network virtualization.
> >> It does not have required ecosystem support to be a viable option and
> > [PG] What is the advantage of moving from NVGRE to GRE-in-UDP, as
> opposed to moving to VXLAN? VXLAN has much better ecosystem support
> so one would immediately benefit from that, no? Outside of network
> virtualization GRE-in-UDP may have use cases and I have never questioned
> those as I am not familiar with those.
> mechanism of extensibility for a low level protocol. In the end, both NVGRE
> and VXLAN suffer from the same limitation in lack of extensibility (VXLAN
> more than GRE), but NVGRE over UDP seems reasonable to at least define if
> there is a deployed base of NVGRE that could benefit.
> 
[PG] From my NVGRE experience in large datacenters and enterprise market, I am not sure of any benefits from GRE-in-UDP. If we need better entropy, we would go with VXLAN and if we need a future encap, we would go with either VXLAN-GPE+NSH or Geneve. There is no point in picking GRE-in-UDP and then waiting for NIC and switch ecosystem around it to turn up, only to then realize that it doesn't offer required extensibility. So not really, I don't see value in GRE-in-UDP for network virtualization.

> >> NVGRE is widely used in datacenter networks and there is wide
> >> hardware support available for it (both in terms of NICs and
> >> Switches). The key advantage VXLAN has over NVGRE is that it allows
> >> larger entropy and doesn't require any change in middle boxes to
> >> calculate hash and do ECMP. Most network virtualization solutions are
> >> supporting both of these today. Even though in long term, things may
> >> settle on a single encap format, that is not practically possible in
> >> the short term. In my opinion, both NVGRE and VXLAN are here to stay
> >> for quite a while given the investment and usage of these
> >> technologies. Could we have settled on just one? Yes, we could have, but
> I am not sure either of these would be the one (read further).
> >>
> >> Shortcomings:
> >> encapsulation. It seems other people were thinking of the same at the
> same time. This resulted in 3 encap options for "future"
> >> more importantly I feel they offer advantage over GUE. VXLAN-GPE +
> >> NSH seems to be the most flexible of the present options as it allows
> offloads.
> >> useful) to have hardware participate. The key limitation of TLV
> >> format (that I have heard of) is that it is harder to parse in
> >> hardware at line rate. This is where I think NSH has an advantage.
> >> NSH allows us to define new MDType so that we can create fixed format
> >> headers (that are easier for hardware to parse than TLV). This allows
> >> faster innovation in software (using vendor specific TLV) and later
> >> on they can be converted into hardware friendly format using MDType.
> >>
> >> GRE-in-UDP
> >> I am not really sure where this fits in for network virtualization.
> >> It does not have required ecosystem support to be a viable option and
> >> > > (in terms of new encap, new numbers, etc, etc!) and a lot of
> >> > > extra work for implementors and users alike! Just carrying GRE
> >> > > over UDP (could be thought of as a nested encap) along with using
> >> > > a few reserved bits (for OAM, etc) seems to me to give all that
> >> > > we’ll get after a long arduous journey to VXLAN GPE through
> >> > > VXLAN! May be my personal
> >> > opinion
> >> > >>> specific for Ethernet/IP payload. We should not make them as
> >> > >>> two distinct encap. protocols for this space. NVGRE can be
> >> > >>> benefit from GRE/UDP for flow entropy and UDP checksum in IPv6;
> >> > >>> thus should be considered as optional to use.
> >> > >>>
> >> > >>> The usage of the UDP header in such encapsulations is mostly
> >> > >>> for entropy (load-balancing), aid NAT traversal, etc. This
> >> > >>> when needed - the destination UDP port, as usual, identifies
> >> > >>> the encap protocol. Given that GRE is, by far, the most
> >> > >>> commonly used general purpose encap deployed today, it may be
> >> > >>> good to have GRE in UDP. I would like to see this as GRE in UDP
> >> > >>> rather than GRE and
> >> > >>>> In my personal opinion, for network virtualization, I don't
> >> > >>>> see much value in GRE in UDP. Each new encapsulation takes a
> >> > >>>> lot of effort and time before it achieves ecosystem support
> >> > >>>> (from software to hardware to
> >> > >>>> offloads) to make it a viable option. We already have
> >> > >>>> VXLAN/GPE so I am not sure what we gain with GRE in UDP over
> >> > >>>> VXLAN/VXLAN-
> >> GPE.
> >> > >>>> There may be scenarios outside of network virtualization,
> >> > >>>> where this may be useful, but I am not familiar with those.
> >> > >>>>
> >> > >>>> Thanks
> >> > >>>> Pankaj
> >> > >>>>
> >> > >>>>>>> https://na01.safelinks.protection.outlook.com/?url=rfc-
> editor.
> >> > >>>>>>> or
> >> > >>>>>>> g&
> >> > >>>>>>> dat
> >> > >>>>>>>
> >> > >>>>>
> >> >
> >>
> >> > >>>>>>> https://na01.safelinks.protection.outlook.com/?url=rfc-
> editor.
> >> > >>>>>>> or
> >> > >>>>>>> g&
> >> > >>>>>>> dat
> >> > >>>>>>>
> >> > >>>>>
> >> >
> >>
> >> > >>>>>>> https://na01.safelinks.protection.outlook.com/?url=rfc-
> editor.
> >> > >>>>>>> or
> >> > >>>>>>> g&
> >> > >>>>>>> dat
> >> > >>>>>>>
> >> > >>>>>
> >> >
> >>
> >> > >>>>>>> multi-tenant data centers.  Network Virtualization
> >> > >>>>>>> decouples virtual networks and addresses from physical
> >> > >>>>>>> network infrastructure, providing isolation and concurrency
> >> > >>>>>>> between multiple virtual networks on the same physical
> >> > >>>>>>> network infrastructure.  This document also introduces a
> >> > >>>>>>> Network Virtualization framework to illustrate the use
> >> > >>>>>>> cases, but the focus is on specifying the data- plane
> >> > >>>>>>> aspect
> >> > >>>>> of NVGRE.
> >> > >>>>>>>
> >> > >>>>>>>
> >> > w

---

From: Tom Herbert <[email-masked]>
Date: Thu, 29 Oct 2015 22:32:57 -0700

> [PG] We have not tested it but maybe some IHVs who are supporting Geneve (or NSH) offload can comment. I have no technical evidence, though, as to why it would _not_ work (sans some implementation…

The technical evidence can be found in evaluating mechanisms of
similar protocols at the same layer. In particular, Geneve TLVs like
IP options and NSH is similar to IP extension headers. Both IP options
and extension headers are currently undeployable in the data center
due to insufficient or incorrect HW support in switches an NICs, and I
haven't (yet) seen a reasonable explanation or implementation showing
that the fate of Geneve TLVs or NSH will be any different.

GUE follows the model of extensibility of GRE. In developing GUE, we
already had a lot of deployment experience with GRE fields (keyid,
csum, seq). The various HW we tested had no issues with them, so we do
believe GUE extensions can be deployed at scale.

Anyway, the model of extensibility is a significant differentiator in
the three NVo3 protocols. I hope there will be some detailed
discussion around this at some point!

Thanks,
Tom

---

From: Dino Farinacci <[email-masked]>
Date: Fri, 30 Oct 2015 00:04:35 -0700

Pankaj, you have shown precisely the point I have been hounding this mailing list. There are just way too many combinations of encapsulation solutions with little or no *real* incremental benefit.

Enough with the data-planes. Once they are implemented and people stop the urge to keep changing something so basic, we can get on with what really matters in building data center overlay networks. That is the control-plane. That will require a lot more thought, experimentation, experience, and deployment.

Dino

> On Oct 29, 2015, at 12:10 PM, Pankaj Garg <[email-masked]> wrote:
> 
> NVGRE and VXLAN

---

From: Pankaj Garg <[email-masked]>
Date: Fri, 30 Oct 2015 17:52:02 +0000

Inline.

> Sent: Thursday, October 29, 2015 10:33 PM
> The technical evidence can be found in evaluating mechanisms of similar
> protocols at the same layer. In particular, Geneve TLVs like IP options and
> NSH is similar to IP extension headers. Both IP options and extension
> headers are currently undeployable in the data center due to insufficient or
> incorrect HW support in switches an NICs, and I haven't (yet) seen a
> reasonable explanation or implementation showing that the fate of Geneve
> TLVs or NSH will be any different.
[PG] I think you are mixing two things hardware and software extensibility. TLV mechanism is perfect for software extensibility. For hardware extensions (as and when they are needed), we can define new MDType that uses fixed format headers to provide hardware interop or having first TLV as hardware extension TLV etc. This is unlike IP options or extension headers. I believe Intel has already implemented offload support for Geneve so maybe they can share their performance numbers but if I recall right (and maybe I am wrong), it is similar to VXLAN performance. So no, I don't think there is any technical evidence against using TLVs for extensions for the purpose of network virtualization.
> already had a lot of deployment experience with GRE fields (keyid, csum,
> seq). The various HW we tested had no issues with them, so we do believe
> GUE extensions can be deployed at scale.
> 
[PG] I don't think GUE provides flexibility that is needed for future encapsulation. Network virtualization is mostly used with-in datacenters and in such environments, flexibility is needed to change and innovate rapidly. We need an encapsulation format that provides such flexibility and does not tie our hands.
> Anyway, the model of extensibility is a significant differentiator in the three
> NVo3 protocols. I hope there will be some detailed discussion around this at
> some point!
[PG] Yes, we should have a discussion on this and I _really_ hope, we pick one protocol as standard for network virtualization as opposed to 3 or 4.

---

From: Pankaj Garg <[email-masked]>
Date: Fri, 30 Oct 2015 17:56:51 +0000

I agree and I have said the same thing in last IETF meeting I attended that right now there are just too many encap protocols without any major benefits over one another. After experience with NVGRE and VXLAN, I hope, we all know and understand to decide which encapsulation would meet various requirements, both from hardware and software perspective. I really hope we pick one encap as standard NVO3 encap as opposed to multiple.

I also agree that control plane is more interesting side to focus on, given the diversity and lack of any common standard to provision things. Right now, we have to live with vendor specific methods, which are painful to manage.

Pankaj

> Sent: Friday, October 30, 2015 12:05 AM
> Cc: Manish Kumar (manishkr) <[email-masked]>; Tom Herbert
> <[email-masked]>; Lucy Yong <[email-masked]>;
> [email-masked]
> Pankaj, you have shown precisely the point I have been hounding this
> mailing list. There are just way too many combinations of encapsulation
> solutions with little or no *real* incremental benefit.
> 
> Enough with the data-planes. Once they are implemented and people stop
> the urge to keep changing something so basic, we can get on with what really
> matters in building data center overlay networks. That is the control-plane.
> That will require a lot more thought, experimentation, experience, and
> deployment.
> 
> Dino
> 
> > Out of the three, I personally prefer the first two, partly because I have
> been deeply involved in the extensibility design of these but more
> importantly I feel they offer advantage over GUE. VXLAN-GPE + NSH seems
> to be the most flexible of the present options as it allows extensibility in
> hardware and software friendly manner. When we think about extensibility,
> there is always struggle between hardware and software. While software
> can evolve much faster, hardware cannot and we have to design something
> that does not restrict software to innovate but eventually let hardware catch
> up and use those innovations as well.
> >
> protocols without breaking NIC offloads. This is a huge advantage given that
> in a network virtualization environment, typically most endpoints are
> software. This allows software vendors to do faster innovation without
> worrying about the whole ecosystem aspects of it. Both Geneve and NSH
> provide this extensibility using TLV format and they both use exact same TLV
> format (by choice).
> >
> may

---

From: "Fedyk, Don" <[email-masked]>
Date: Fri, 30 Oct 2015 18:07:11 +0000

+1  Although the dream to have "One Encap to Unite them all"  may not be achievable within this Realm :-)

Don 

> From: nvo3 [mailto:[email-masked]] On Behalf Of Dino Farinacci
> Sent: Friday, October 30, 2015 3:05 AM
> To: Pankaj Garg
> Cc: Tom Herbert; [email-masked]; Manish Kumar (manishkr); Lucy Yong

---

From: Tom Herbert <[email-masked]>
Date: Fri, 30 Oct 2015 11:57:15 -0700

> [PG] I don't think GUE provides flexibility that is needed for future encapsulation. Network virtualization is mostly used with-in datacenters and in such environments, flexibility is needed to cha…

Well, we have already defined seven extensions to GUE. AFAICT adding
these was quite straightforward none of these can break forward
compatibility, nor NIC offloads, and is amenable to efficient header
parsing in both software and hardware. GUE also allows for private
data in the header section for a site or application to insert data
with whatever format is suitable (for instance, if SPUD uses GUE
format CBOR data could go here).  But, if you really do see an
deficiency in this model that would "tie your hands" please elaborate,
we appreciate the feedback!

Thanks,
Tom

---

From: Pankaj Garg <[email-masked]>
Date: Fri, 30 Oct 2015 20:38:35 +0000

Inline.

> Sent: Friday, October 30, 2015 11:57 AM
> > [PG] I don't think GUE provides flexibility that is needed for future
> encapsulation. Network virtualization is mostly used with-in datacenters and
> in such environments, flexibility is needed to change and innovate rapidly.
> We need an encapsulation format that provides such flexibility and does not
> tie our hands.
> 
> compatibility, nor NIC offloads, and is amenable to efficient header parsing in
> both software and hardware. GUE also allows for private data in the header
> section for a site or application to insert data with whatever format is suitable
> (for instance, if SPUD uses GUE format CBOR data could go here).  But, if you
> really do see an deficiency in this model that would "tie your hands" please
> elaborate, we appreciate the feedback!
[PG] Our network stack consists of multiple layers, where layers can be developed independently (and can even be from separate vendors). Using Geneve/NSH style TLV provides us flexibility of a non-conflicting private data space, where different layers can insert their own data on transmit and process that on reception. How would one achieve this flexibility in GUE?

---

From: Tom Herbert <[email-masked]>
Date: Fri, 30 Oct 2015 14:48:26 -0700

> [PG] Our network stack consists of multiple layers, where layers can be developed independently (and can even be from separate vendors). Using Geneve/NSH style TLV provides us flexibility of a non-…

You can define proprietary options (TLVs or other format) in the
private data space. The interpretation of the this space is agreement
between the sender and the receiver so there should be no ambiguity or
limits to what can be placed there (subject only to header size
constraints).

Tom

---

From: Pankaj Garg <[email-masked]>
Date: Fri, 30 Oct 2015 21:54:39 +0000

Inline.

> Sent: Friday, October 30, 2015 2:48 PM
> >> encapsulation. Network virtualization is mostly used with-in
> >> datacenters and in such environments, flexibility is needed to change and
> innovate rapidly.
> >> We need an encapsulation format that provides such flexibility and
> >> does not tie our hands.
> >>
> >> deficiency in this model that would "tie your hands" please elaborate, we
> appreciate the feedback!
> > [PG] Our network stack consists of multiple layers, where layers can be
> developed independently (and can even be from separate vendors). Using
> Geneve/NSH style TLV provides us flexibility of a non-conflicting private data
> space, where different layers can insert their own data on transmit and
> process that on reception. How would one achieve this flexibility in GUE?
> 
> You can define proprietary options (TLVs or other format) in the private data
> space. The interpretation of the this space is agreement between the sender
> and the receiver so there should be no ambiguity or limits to what can be
> placed there (subject only to header size constraints).
[PG] Yes, which is what TLVs in NSH/Geneve do but these are part of the format and not something we have to define on the side. Two independent entities can attach their metadata on the same packet without conflicts etc. Eventually, one can take either of these encap protocols and twist/turn them to achieve what is needed, e.g. in Geneve one can define a standard first TLV that has flag based options etc. as well. In GUE, I can create another IETF draft to define "standard" TLV based usage of "private" data space. That is why, I think many people on the alias feel that many of these encap don't have any substantial value over one another, other than semantic differences. That said, out of the box, IMHO, NSH has the most flexibility i.e. define new MDType, using that MDType for flag based options in fixed header and TLVs for variable length.

---

From: Dino Farinacci <[email-masked]>
Date: Fri, 30 Oct 2015 16:13:55 -0700

Thanks for the response Pankaj.

> On Oct 30, 2015, at 10:56 AM, Pankaj Garg <[email-masked]> wrote:
> 

Are the chairs listening? Can we force a decision?

> I also agree that control plane is more interesting side to focus on, given the diversity and lack of any common standard to provision things. Right now, we have to live with vendor specific method…

We need network elements to talk to each other. In the pre-SDN error, we built things called protocols. We should do that now and have SDN-like things manage the protocols as a separate task (that is going to happen anyways).

Dino

---

From: Dino Farinacci <[email-masked]>
Date: Fri, 30 Oct 2015 16:14:41 -0700

But not talking about one assures there will never be one.

Dino

> On Oct 30, 2015, at 11:07 AM, Fedyk, Don <[email-masked]> wrote:
>

---

From: Tom Herbert <[email-masked]>
Date: Fri, 30 Oct 2015 16:25:42 -0700

> [PG] Yes, which is what TLVs in NSH/Geneve do but these are part of the format and not something we have to define on the side. Two independent entities can attach their metadata on the same packet…

What I meant to say is that the format of private data could be typed
in GUE via an option. For instance, we could define a data format for
Geneve and then, presto, GUE can carry Geneve TLVs. While personally I
am very leery about deploying any new use cases of TLVs in my data
center (given the lessons learned with IP options and extension
headers), if allowing them in GUE would achieve a compromise that
pulls us closer to defining _one_ NVo3 protocol, then it seems like a
reasonable direction.

Tom

---

From: Pankaj Garg <[email-masked]>
Date: Sat, 31 Oct 2015 00:17:56 +0000

Inline.
> Sent: Friday, October 30, 2015 4:26 PM
> > [PG] Yes, which is what TLVs in NSH/Geneve do but these are part of the
> format and not something we have to define on the side. Two independent
> entities can attach their metadata on the same packet without conflicts etc.
> Eventually, one can take either of these encap protocols and twist/turn them
> to achieve what is needed, e.g. in Geneve one can define a standard first TLV
> that has flag based options etc. as well. In GUE, I can create another IETF
> draft to define "standard" TLV based usage of "private" data space. That is
> why, I think many people on the alias feel that many of these encap don't
> have any substantial value over one another, other than semantic
> differences. That said, out of the box, IMHO, NSH has the most flexibility i.e.
> define new MDType, using that MDType for flag based options in fixed
> header and TLVs for variable length.
> 
> What I meant to say is that the format of private data could be typed in GUE
> via an option. For instance, we could define a data format for Geneve and
> then, presto, GUE can carry Geneve TLVs. While personally I am very leery
> about deploying any new use cases of TLVs in my data center (given the
> lessons learned with IP options and extension headers), if allowing them in
> GUE would achieve a compromise that pulls us closer to defining _one_
> NVo3 protocol, then it seems like a reasonable direction.
[PG] Yes, if there is a standard option to carry TLVs then it would meet the flexible extension requirement and make it similar to Geneve and NSH from flexible extension perspective. In terms of industry adoption, there may be other factors like hardware and software ecosystem support, but from purely format perspective, it would certainly make GUE meeting flexibility requirements better.

---

From: Jesse Gross <[email-masked]>
Date: Sat, 31 Oct 2015 01:04:22 +0000

From: nvo3 on behalf of Pankaj Garg
Date: Saturday, October 31, 2015 at 8:17 AM
To: Tom Herbert
Cc: "[email-masked]<mailto:[email-masked]>", "Manish Kumar (manishkr)", Dino Farinacci, Lucy yong
Subject: Re: [nvo3] RFC 7637 on NVGRE: Network Virtualization Using Generic Routing Encapsulation

Inline.
-----Original Message-----
From: Tom Herbert [mailto:[email-masked]]
Sent: Friday, October 30, 2015 4:26 PM
To: Pankaj Garg <[email-masked]<mailto:[email-masked]>>
Cc: Dino Farinacci <[email-masked]<mailto:[email-masked]>>; Manish Kumar (manishkr)
<[email-masked]<mailto:[email-masked]>>; Lucy Yong <[email-masked]<mailto:[email-masked]>>;
[email-masked]<mailto:[email-masked]>
Subject: Re: [nvo3] RFC 7637 on NVGRE: Network Virtualization Using
Generic Routing Encapsulation

To follow up on Pankaj’s mention of ecosystem support, one comment about the viability of TLVs is that whether they are a useful extension mechanism is mostly based on the implementer’s perception. If they are seen as an add-on that is not really core functionality (as in IPv4 and IPv6), then sure, people won’t bother to support them. However, in the case of Geneve, they are obviously the major goal of the protocol and they are being implemented, in both software and hardware.

As a result, it’s not an inherent property of TLVs that they are implemented or not. In order to make them useful, they need to be used and considered important from day 1. Making them available as an additional extensibility mechanism pretty much guarantees that they won’t be available in the future, as we’ve seen.

---

From: Xuxiaohu <[email-masked]>
Date: Sat, 31 Oct 2015 01:06:41 +0000

> Sent: Friday, October 30, 2015 3:05 PM
> To: Pankaj Garg
> Cc: Tom Herbert; [email-masked]; Manish Kumar (manishkr); Lucy yong
> Pankaj, you have shown precisely the point I have been hounding this mailing
> list. There are just way too many combinations of encapsulation solutions with
> little or no *real* incremental benefit.
> 
> Enough with the data-planes. Once they are implemented and people stop the
> urge to keep changing something so basic, we can get on with what really
> matters in building data center overlay networks. That is the control-plane. That

Since we have learnt enough about what REALLY matters in building data center overlay network (i.e., DC VPNs), it's the time now for the WG to recommend one or two data-plane solutions among so many options.

Best regards,
Xiaohu 

> will require a lot more thought, experimentation, experience, and deployment.
> 
> Dino
> 
> support available for it (both in terms of NICs and Switches). The key advantage
> VXLAN has over NVGRE is that it allows larger entropy and doesn't require any
> change in middle boxes to calculate hash and do ECMP. Most network
> virtualization solutions are supporting both of these today. Even though in long
> term, things may settle on a single encap format, that is not practically possible
> in the short term. In my opinion, both NVGRE and VXLAN are here to stay for
> quite a while given the investment and usage of these technologies. Could we
> have settled on just one? Yes, we could have, but I am not sure either of these
> would be the one (read further).
> >
> > Shortcomings:
> one severe limitation (well VXLAN had two) i.e. ability (or lack thereof) to extend
> them. In addition, VXLAN even did not allow encap of non-ethernet inner packet.
> The main problem for network virtualization, however was extensibility. There
> are a lot of use cases where such extensibility can add significant advantage to
> encapsulation. It seems other people were thinking of the same at the same time.
> This resulted in 3 encap options for "future" encap. VXLAN-GPE + NSH, Geneve
> and GUE.
> >
> deeply involved in the extensibility design of these but more importantly I feel
> they offer advantage over GUE. VXLAN-GPE + NSH seems to be the most flexible
> of the present options as it allows extensibility in hardware and software friendly
> manner. When we think about extensibility, there is always struggle between
> hardware and software. While software can evolve much faster, hardware
> cannot and we have to design something that does not restrict software to
> Both Geneve and VXLAN-GPE+NSH allows extension of these protocols without
> breaking NIC offloads. This is a huge advantage given that in a network
> virtualization environment, typically most endpoints are software. This allows
> software vendors to do faster innovation without worrying about the whole
> ecosystem aspects of it. Both Geneve and NSH provide this extensibility using
> TLV format and they both use exact same TLV format (by choice).
> >
> > However, not all endpoints can be software and eventually for some of those
> innovations (such as security or OAM etc.) it would be required (or useful) to
> have hardware participate. The key limitation of TLV format (that I have heard of)
> is that it is harder to parse in hardware at line rate. This is where I think NSH has
> an advantage. NSH allows us to define new MDType so that we can create fixed
> format headers (that are easier for hardware to parse than TLV). This allows
> faster innovation in software (using vendor specific TLV) and later on they can
> be converted into hardware friendly format using MDType.
> >
> > GRE-in-UDP
> have required ecosystem support to be a viable option and does not solve the
> need for future encapsulations.
> >
> "future" encap as the standard option to avoid unnecessary fragmentation and
> engineering overhead.
> >
> > Thanks
> > Pankaj
> >
> >
> >
> oft.com%7cb4e8bc27b4104053ac7d08d2ca97cf9d%7c72f988bf86f141af91ab
> 2
> >>>>>>> d
> >>>>>>> 7c
> >>>>>>>>>
> >>>>>>>
> >>
> 40microsoft.com%7cb4e8bc27b4104053ac7d08d2ca97cf9d%7c72f988bf86f1
> 4
> >>>>>>> 1
> >>>>>>> af
> >>>>>>>>>
> >>>>>>>
> >>
> 7cb4e8bc27b4104053ac7d08d2ca97cf9d%7c72f988bf86f141af91ab2d7cd01
> 1d
> >>>>>>> b
> >>>>>>>
> >>
> b4e8bc27b4104053ac7d08d2ca97cf9d%7c72f988bf86f141af91ab2d7cd011d
> b4
> >>>>>>> 7

---

From: Tom Herbert <[email-masked]>
Date: Fri, 30 Oct 2015 18:40:03 -0700

> To follow up on Pankaj’s mention of ecosystem support, one comment about the
> viability of TLVs is that whether they are a useful extension mechanism is
> mostly based on the implementer’s perception. If they are seen as an add-on
> that is not really core functionality (as in IPv4 and IPv6), then sure,
> people won’t bother to support them. However, in the case of Geneve, they
> are obviously the major goal of the protocol and they are being implemented,
> in both software and hardware.
>
Jesse,

Your point that TLVs are major goal of Geneve would be much stronger
if you could reference defined TLVs that are critical to the protocol
function. Maybe I'm missing something, but I can't find any defined
TLVs for Geneve on the web at all.

As for efficient hardware implementation, section 2.2.1 of the Geneve
draft does acknowledge the problem for VLH at least:

"The use of a variable length header and options in a protocol often
raises questions about whether it is truly efficiently implementable
in hardware."

Unfortunately, the rest of the section offers no practical guidance to
HW or SW vendors on how to do efficient implementation.  The closest
thing to a mitigation for the potential performance problem is:

"As new functionality becomes sufficiently well defined to add to
endpoints, supporting options can be designed using ordering
restrictions and other techniques to ease parsing."

This is obviously vague. I suppose the ordering restrictions are the
proposed solution for the combinatorial parsing problem of TLVs. But
without details on how there is no teeth behind this idea. As for
these "other techniques"...????

> As a result, it’s not an inherent property of TLVs that they are implemented
> or not. In order to make them useful, they need to be used and considered
> important from day 1. Making them available as an additional extensibility
> mechanism pretty much guarantees that they won’t be available in the future,
> as we’ve seen.

Put the VNID into a TLV then you are guaranteed that people will implement them!

Tom

---

From: Anoop Ghanwani <[email-masked]>
Date: Fri, 30 Oct 2015 18:42:36 -0700

On Fri, Oct 30, 2015 at 6:40 PM, Tom Herbert <[email-masked]> wrote:

> And don't forget to make the VNID variable length while we're at it!!

---

From: Tom Herbert <[email-masked]>
Date: Fri, 30 Oct 2015 19:00:40 -0700

> And don't forget to make the VNID variable length while we're at it!!
+1

---

From: Anoop Ghanwani <[email-masked]>
Date: Fri, 30 Oct 2015 19:47:26 -0700

On Fri, Oct 30, 2015 at 7:00 PM, Tom Herbert <[email-masked]> wrote:

> >> Put the VNID into a TLV then you are guaranteed that people will
> implement
> >> them!
> >>

Just to be clear I just threw that out in jest (I know I should be more
careful about my jokes).  That said, I know it's been discussed that 24
bits, which many of the proposals seem to use, may not be enough.  I am
sympathetic to that, but I'm also sympathetic to ASIC designers being able
to get these proposals implemented.

I do like the discussion about implementation in Appendix D of the GUE
draft.  The problem then becomes one of determining what should be
supported in the fast path; once we have done that, we effectively have
defined a fixed-header solution.  And if we don't do that, we risk lack of
interoperability as each vendor independently tries to guess what that set
should be.

Anoop

---

From: Jesse Gross <[email-masked]>
Date: Sat, 31 Oct 2015 04:29:05 +0000

From: Tom Herbert
Date: Saturday, October 31, 2015 at 9:40 AM
To: Jesse Gross
Cc: Pankaj Garg, "[email-masked]<mailto:[email-masked]>", "Manish Kumar (manishkr)", Dino Farinacci, Lucy yong
Subject: Re: [nvo3] RFC 7637 on NVGRE: Network Virtualization Using Generic Routing Encapsulation

Jesse,

Here is an example from OVN:
https://github.com/openvswitch/ovs/blob/master/ovn/ovn-architecture.7.xml#L1014

OVN is actually a pretty useful reference in general because it is an open source network virtualization implementation that is using Geneve as its preferred encapsulation.

And here is a list of implementations of Geneve that I am aware of, from my presentation at the meeting in Prague plus a few additional ones that I became aware of since then. We don’t really need to speculate what people might do, given that there are a number of implementations already from different vendors. (This is all public.):

Controller:
Open Virtual Networking (OVN)

Software Endpoint:
Open vSwitch
Linux

Debugging Tool:
Wireshark
tcpdump
libpcap

NIC:
Intel XL710
Mellanox ConnectX-4
Broadcom NetXtreme C-Series
QLogic 578xx
Netronome NFP-6xxx

Switching ASIC:
Broadcom Trident 2+/DNX
Cavium XPliant
Mellanox Spectrum
Intel Red Rock Canyon
Centec GoldenGate
Marvell Prestera

---

From: Diego Garcia del Rio <[email-masked]>
Date: Sat, 31 Oct 2015 19:40:01 -0700

Thanks for the references. But out of all the switching asics, which are
arguably the most constrained in terms parsing flexibility and need for
performance, how many support arbitrary tlvs push/pop? Geneve in its most
basic form without options looks very similar to vxlan and thus I'm not
surprised we can see several of the products claiming support for it. But
when we start adding arbitrary protocol types and variable options I'm not
so sure that those implementations can claim full support. Even for the
nics and all the offloads, it comes a moment where the parsing engines
can't look deep enough into the packet and then we have a problem.

I think there is a reason why there is quite some pushback from the
hardware side against variable length options.

My two cents,
Diego

On Oct 30, 2015, at 21:29, Jesse Gross <[email-masked]> wrote:

From: Tom Herbert
Date: Saturday, October 31, 2015 at 9:40 AM
To: Jesse Gross
Cc: Pankaj Garg, "[email-masked]", "Manish Kumar (manishkr)", Dino
Farinacci, Lucy yong
Subject: Re: [nvo3] RFC 7637 on NVGRE: Network Virtualization Using Generic
Routing Encapsulation

Jesse,

OVN is actually a pretty useful reference in general because it is an open
source network virtualization implementation that is using Geneve as its
preferred encapsulation.

And here is a list of implementations of Geneve that I am aware of, from my
presentation at the meeting in Prague plus a few additional ones that I
became aware of since then. We don’t really need to speculate what people
might do, given that there are a number of implementations already from
different vendors. (This is all public.):

Controller:

Software Endpoint:
Open vSwitch
Linux

Debugging Tool:
Wireshark
tcpdump
libpcap

NIC:
Intel XL710
Mellanox ConnectX-4

Switching ASIC:
Broadcom Trident 2+/DNX
Cavium XPliant
Mellanox Spectrum
Intel Red Rock Canyon
Centec GoldenGate
Marvell Prestera

---

From: Alia Atlas <[email-masked]>
Date: Sun, 1 Nov 2015 00:03:15 -0400

I am delighted to see this discussion about evaluating the benefits and
concerns on the different encapsulations.

I would really prefer to see the NVO3 WG to have some serious discussion
at a minimum.

Regards,
Alia

On Fri, Oct 30, 2015 at 1:32 AM, Tom Herbert <[email-masked]> wrote:

> > [PG] We have not tested it but maybe some IHVs who are supporting Geneve
> (or NSH) offload can comment. I have no technical evidence, though, as to
> why it would _not_ work (sans some implementation constraints on maximum
> header size etc, which is pretty reasonable).
>

---

From: Jesse Gross <[email-masked]>
Date: Sun, 1 Nov 2015 04:12:21 +0000

Sure, probably all of the hardware implementations have some limits on their ability to handle the full breadth of Geneve options. Geneve was intentionally designed to be very future proof and support limits beyond what I would realistically expect people to implement/use in the near future. The more interesting question is whether it is possible to support a useful set for what we need today. Unfortunately, I can’t really give specifics for different implementations (in the cases that I know) but let me try to make some generalizations and you can follow up with individual vendors if you need more details.

NICs generally don’t have much of an issue and seem to have largely settled on supporting 64 bytes of options with offloads for the current generation. I don’t really see it as an issue if the parser can’t looker deeper than this. Assuming that 64 option bytes is enough for data packets, the control plane would simply configure senders to not use more than that. And in the edge cases where we need to use more than that (perhaps a control packet), the packets would still flow, just without using offloads.

Switching ASICs are more complicated and varied, as you say. There are definitely implementations that can support parsing and generating a set of TLVs at full multi-terabit data rates (16 or 32 bytes are popular here) and undoubtedly some that are just supporting VXLAN-like capabilities with a few additional features. Even the latter is useful since it allows transitioning to a single protocol with incremental use of additional functionality, especially since in some cases this can be enabled with existing chips that are already deployed.

One final point – most of these limitations are around the length of the header, not the variable nature of it. So even if Geneve as a protocol is capable of supporting more than what is used today, the baseline is probably pretty similar to the alternatives and the future flexibility is hugely useful. Without a doubt, implementing variable length headers is more effort than fixed length but based on the above implementations, it seems clearly doable if the benefit is there. Given the number of extensions that we have seem to VXLAN, this type of extensibility seems to be essential.

Jesse

From: Diego Garcia del Rio
Date: Sunday, November 1, 2015 at 11:40 AM
To: Jesse Gross
Cc: Tom Herbert, Pankaj Garg, "[email-masked]<mailto:[email-masked]>", "Manish Kumar (manishkr)", Dino Farinacci, Lucy yong
Subject: Re: [nvo3] RFC 7637 on NVGRE: Network Virtualization Using Generic Routing Encapsulation

Thanks for the references. But out of all the switching asics, which are arguably the most constrained in terms parsing flexibility and need for performance, how many support arbitrary tlvs push/pop? Geneve in its most basic form without options looks very similar to vxlan and thus I'm not surprised we can see several of the products claiming support for it. But when we start adding arbitrary protocol types and variable options I'm not so sure that those implementations can claim full support. Even for the nics and all the offloads, it comes a moment where the parsing engines can't look deep enough into the packet and then we have a problem.

I think there is a reason why there is quite some pushback from the hardware side against variable length options.

My two cents,
Diego

On Oct 30, 2015, at 21:29, Jesse Gross <[email-masked]<mailto:[email-masked]>> wrote:

From: Tom Herbert

Jesse,

Controller:

Software Endpoint:
Open vSwitch
Linux

Debugging Tool:
Wireshark
tcpdump
libpcap

NIC:
Intel XL710
Mellanox ConnectX-4

Switching ASIC:
Broadcom Trident 2+/DNX
Cavium XPliant
Mellanox Spectrum
Intel Red Rock Canyon
Centec GoldenGate
Marvell Prestera