Subject: Re: [nvo3] Extensions in nvo3 encapsulations

From: Lizhong Jin <[email-masked]>
Date: Fri, 19 Aug 2016 23:07:12 +0800

Hi Tom,
Thanks for the information. This gives me an overall view of the
extensions. I try to understand, and see inline below.

Lizhong

---------- Forwarded message ----------
> From: Tom Herbert <[email-masked]>
> To: "[email-masked]" <[email-masked]>
> Cc:
> Date: Thu, 18 Aug 2016 16:24:42 -0700
> Subject: [nvo3] Extensions in nvo3 encapsulations
> Alia has asked that we provide some more information as to what the
> motivations for extensions are for nvo3 and to what form they should
> take. Here is some high level explanation with respect to GUE.
>
> Six fundamental extensions have been defined for GUE
>
> 1) VNID (draft-hy-nvo3-gue-4-nvo-03)
> 2) Checksum (draft-herbert-gue-extensions-00)
>
[Lizhong] one more checksum will add additional load to hardware NAT, and
it is required to update checksum for every NAT router which was not
supposed to support GUE.

> 3) Remote checksum offload or RCO  (draft-herbert-gue-extensions-00)
>
[Lizhong] this is a feature to enhance legacy NIC. But is two checksum
(e.g., both outer&inner UDP checksum enabled) enabled really useful? It
seems, to me, only outer or inner UDP checksum is enough.

> 4) Fragmentation (draft-herbert-gue-extensions-00)
>
[Lizhong] such kind of fragmentation is also not perfect. E.g., the
receiver side hardware without reassembling could not get inner header, and
fail to do RSS. I still like #3 as described in the draft.

> 5) Security (draft-herbert-gue-extensions-00)
> 6) Payload transform (draft-herbert-gue-extensions-00)
>
[Lizhong] I doubt if GUE header security is necessary. In MPLS network, we
also use label to implicitly identify a VPN, and it works well.

> All of these share some common properties:
>
> * The extensions seem cover three areas: integrity/security,
> addressing, and protocol processing.
> * These extensions break down into two classes: those that can be
> processed without any additional information (checksum, RCO,
> fragmentation), and those that require context (VNID, security,
> payload xfrm)..
> * These are all intrinsic to the encapsulation layer and are part of
> correct processing of the encapsulation when present. For instance the
> security option is expressly needed to secure the GUE header and in
> particular to provide security for VNID.
> * All of these are mandatory in the sense that they can't be ignored
> at a receiver. For instance, if some receiver decided to ignore RCO
> then that would just become a checksum error at the ultimate receiver.
> Anything to do with security can pretty much never be ignored. Also,
> as a general principle in large scale DCs, it's much better to drop a
> packet when we're not sure something is correct as opposed to allowing
> it through even if the risk is slight. Packet loss is much easier to
> debug than incorrectness!
> * With the exception of fragmentation, a subset of these will commonly
> be set in packets for some deployment. For instance in a nvo3
> deployment, VNID and security might be set all the time.
> * These are not intended for processing by middleboxes in transit. As
> I mentioned in my response to Bob Briscoe, intermediate devices cannot
> correctly parse UDP payloads (port number ambiguity problem in the
> network). Such information should be in HBH for IPv6 or IP options in
> IPv4.
> * These are interoperable and not proprietary.We allow proprietary
> extensions via a private data field but do do not encourage it, to do
> so otherwise would be to encourage vendor lock-in.
>
> While these six fundamental cover current needs, we cannot predict
> that they cover all the future needs of the protocol. At some point we
> will need to extend a deployed encapsulation protocol in the data
> center. The most likely scenario that we would need to extend the
> protocol is to addressed a newly discovered security vulnerability. We
> anticipate that extending a protocol is a rare event, but at large
> scale we need this capability. Extending a protocol is far simpler
> that replacing a protocol especially if we can avoid needing to swap
> out massive amounts of hardware.
>
> Another general characteristic is that GUE extensions are contained
> within a single header, as opposed to using shim headers or following
> headers. There are benefits to do this:
>
> * It's very clear that the extensions refer to the encapsulation. For
> instance if implement a checksum in a shim header then we need some
> consideration as to how the checksum covers prior headers.
> * We can limit the extensibility. This is important consideration for
> efficiency and avoid DOS attacks.
> * This allows "skip ahead". If a device just wants to look at the
> encapsulated IP packet for instance it does not need to parse a header
> chain for that.
>
> Thanks,
> Tom
>
>
>
>
> ---------- Forwarded message ----------
> From: Lucy yong <[email-masked]>
> To: Tom Herbert <[email-masked]>, "[email-masked]" <[email-masked]>
> Cc:
> Date: Fri, 19 Aug 2016 00:04:55 +0000
> Subject: Re: [nvo3] Extensions in nvo3 encapsulations
> This is very good summary of the motivation for extensions. GUE design was
> given quite thought on that.
>
> For supporting tunnel stitching (draft-yong-nvo3-tunnel-stitching), it
> requires a new field in the encap. protocol to carry the next tunnel
> identifier. GUE can easily support that by defining a new flag and
> corresponding optional field.
>
> Extensibility is important for a protocol to retain long.
>
> Lucy
>
> -----Original Message-----
> From: nvo3 [mailto:[email-masked]] On Behalf Of Tom Herbert
> Sent: Thursday, August 18, 2016 6:25 PM
> To: [email-masked]
> Subject: [nvo3] Extensions in nvo3 encapsulations
>
> Alia has asked that we provide some more information as to what the
> motivations for extensions are for nvo3 and to what form they should take.
> Here is some high level explanation with respect to GUE.
>
> Six fundamental extensions have been defined for GUE
>
> 1) VNID (draft-hy-nvo3-gue-4-nvo-03)
> 2) Checksum (draft-herbert-gue-extensions-00)
> 3) Remote checksum offload or RCO  (draft-herbert-gue-extensions-00)
> 4) Fragmentation (draft-herbert-gue-extensions-00)
> 5) Security (draft-herbert-gue-extensions-00)
> 6) Payload transform (draft-herbert-gue-extensions-00)
>
> All of these share some common properties:
>
> * The extensions seem cover three areas: integrity/security, addressing,
> and protocol processing.
> * These extensions break down into two classes: those that can be
> processed without any additional information (checksum, RCO,
> fragmentation), and those that require context (VNID, security, payload
> xfrm)..
> * These are all intrinsic to the encapsulation layer and are part of
> correct processing of the encapsulation when present. For instance the
> security option is expressly needed to secure the GUE header and in
> particular to provide security for VNID.
> * All of these are mandatory in the sense that they can't be ignored at a
> receiver. For instance, if some receiver decided to ignore RCO then that
> would just become a checksum error at the ultimate receiver.
> Anything to do with security can pretty much never be ignored. Also, as a
> general principle in large scale DCs, it's much better to drop a packet
> when we're not sure something is correct as opposed to allowing it through
> even if the risk is slight. Packet loss is much easier to debug than
> incorrectness!
> * With the exception of fragmentation, a subset of these will commonly be
> set in packets for some deployment. For instance in a nvo3 deployment, VNID
> and security might be set all the time.
> * These are not intended for processing by middleboxes in transit. As I
> mentioned in my response to Bob Briscoe, intermediate devices cannot
> correctly parse UDP payloads (port number ambiguity problem in the
> network). Such information should be in HBH for IPv6 or IP options in IPv4.
> * These are interoperable and not proprietary.We allow proprietary
> extensions via a private data field but do do not encourage it, to do so
> otherwise would be to encourage vendor lock-in.
>
> While these six fundamental cover current needs, we cannot predict that
> they cover all the future needs of the protocol. At some point we will need
> to extend a deployed encapsulation protocol in the data center. The most
> likely scenario that we would need to extend the protocol is to addressed a
> newly discovered security vulnerability. We anticipate that extending a
> protocol is a rare event, but at large scale we need this capability.
> Extending a protocol is far simpler that replacing a protocol especially if
> we can avoid needing to swap out massive amounts of hardware.
>
> Another general characteristic is that GUE extensions are contained within
> a single header, as opposed to using shim headers or following headers.
> There are benefits to do this:
>
> * It's very clear that the extensions refer to the encapsulation. For
> instance if implement a checksum in a shim header then we need some
> consideration as to how the checksum covers prior headers.
> * We can limit the extensibility. This is important consideration for
> efficiency and avoid DOS attacks.
> * This allows "skip ahead". If a device just wants to look at the
> encapsulated IP packet for instance it does not need to parse a header
> chain for that.
>
> Thanks,
> Tom
>
> _______________________________________________
> nvo3 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nvo3
>
>
>
>
> ---------- Forwarded message ----------
> From: Lizhong Jin <[email-masked]>
> To: "[email-masked]" <[email-masked]>
> Cc: Matthew Bocci <[email-masked]>
> Date: Fri, 19 Aug 2016 14:58:53 +0800
> Subject: Re: [nvo3] Call for interest on NVO3 use case draft
> Hi,
> The use case draft did help us a lot at the beginning of NVO3, and will
> also continue to helping other new comers. So I believe it is useful to
> have this draft to be published.
>
> Lizhong
>
>
>
>> ---------- Forwarded message ----------
>> From: "Bocci, Matthew (Nokia - GB)" <[email-masked]>
>> To: NVO3 <[email-masked]>
>> Cc:
>> Date: Fri, 12 Aug 2016 16:27:46 +0000
>> Subject: [nvo3] Call for interest on NVO3 use case draft
>>
>> WG
>>
>>
>>
>> As discussed in Berlin, the NVO3 use cases draft (
>> https://www.ietf.org/id/draft-ietf-nvo3-use-case-08.txt) is reaching
>> maturity and potentially ready for WG last call. However, there have been
>> some questions raised about how useful a use cases draft is at this stage
>> in the working group, and whether some of the content could be more
>> usefully contributed to applicability sections of solutions drafts.
>>
>>
>>
>> Before starting a last call, we would like to ask that if you have read
>> the latest version of the draft, please indicate so to the list, and if so
>> whether you believe we should progress this as a stand-alone draft.
>>
>>
>>
>> This call for interest will end on Friday  26th August.
>>
>>
>>
>> Best regards
>>
>>
>>
>> Matthew
>>
>>
>>
>>
>>
>
>
> ---------- Forwarded message ----------
> From: Patrick Frejborg <[email-masked]>
> To: Tom Herbert <[email-masked]>
> Cc: "[email-masked]" <[email-masked]>
> Date: Fri, 19 Aug 2016 13:39:15 +0300
> Subject: Re: [nvo3] Extensions in nvo3 encapsulations
> Hi Tom,
>
> thinking out of the box - what will break if you slightly change the
> order in which the packets are processed at the VTEP/PE/NVE?
> >From the VM perspective, you insert a new L2 module before the packets
> are sent down to the virtual circuit. In this module you have e.g.
> MACSec for encryption, MEF modules for OAM functions, perhaps add some
> new mechanisms for fragmenting on L2 level , have some kind of Path
> MTU discovery mechanism etc. Depending on you performance requirements
> and amount of tunnels that need these extended features you could run
> this undefined module as VM or when high performance is required then
> have it deployed in DPDK or if hardware appliance exists insert them
> to your hosting device. With service chaining you can direct these
> extended services to appropriate tunnels regardless what kind of L3
> tunnels are deployed between the VTEPs/PEs/NVEs peers as long as the
> remote peer supports this undefined module. If VNID is needed on the
> tunnel itself between the NVE peers, then chose a tunnel protocol that
> supports that, if not then go for the tunnel with the smallest
> overhead, Ethernet over UDP(?) and so on.
>
> Regards,
> Patrick
>
> On Fri, Aug 19, 2016 at 2:24 AM, Tom Herbert <[email-masked]> wrote:
> > Alia has asked that we provide some more information as to what the
> > motivations for extensions are for nvo3 and to what form they should
> > take. Here is some high level explanation with respect to GUE.
> >
> > Six fundamental extensions have been defined for GUE
> >
> > 1) VNID (draft-hy-nvo3-gue-4-nvo-03)
> > 2) Checksum (draft-herbert-gue-extensions-00)
> > 3) Remote checksum offload or RCO  (draft-herbert-gue-extensions-00)
> > 4) Fragmentation (draft-herbert-gue-extensions-00)
> > 5) Security (draft-herbert-gue-extensions-00)
> > 6) Payload transform (draft-herbert-gue-extensions-00)
> >
> > All of these share some common properties:
> >
> > * The extensions seem cover three areas: integrity/security,
> > addressing, and protocol processing.
> > * These extensions break down into two classes: those that can be
> > processed without any additional information (checksum, RCO,
> > fragmentation), and those that require context (VNID, security,
> > payload xfrm)..
> > * These are all intrinsic to the encapsulation layer and are part of
> > correct processing of the encapsulation when present. For instance the
> > security option is expressly needed to secure the GUE header and in
> > particular to provide security for VNID.
> > * All of these are mandatory in the sense that they can't be ignored
> > at a receiver. For instance, if some receiver decided to ignore RCO
> > then that would just become a checksum error at the ultimate receiver.
> > Anything to do with security can pretty much never be ignored. Also,
> > as a general principle in large scale DCs, it's much better to drop a
> > packet when we're not sure something is correct as opposed to allowing
> > it through even if the risk is slight. Packet loss is much easier to
> > debug than incorrectness!
> > * With the exception of fragmentation, a subset of these will commonly
> > be set in packets for some deployment. For instance in a nvo3
> > deployment, VNID and security might be set all the time.
> > * These are not intended for processing by middleboxes in transit. As
> > I mentioned in my response to Bob Briscoe, intermediate devices cannot
> > correctly parse UDP payloads (port number ambiguity problem in the
> > network). Such information should be in HBH for IPv6 or IP options in
> > IPv4.
> > * These are interoperable and not proprietary.We allow proprietary
> > extensions via a private data field but do do not encourage it, to do
> > so otherwise would be to encourage vendor lock-in.
> >
> > While these six fundamental cover current needs, we cannot predict
> > that they cover all the future needs of the protocol. At some point we
> > will need to extend a deployed encapsulation protocol in the data
> > center. The most likely scenario that we would need to extend the
> > protocol is to addressed a newly discovered security vulnerability. We
> > anticipate that extending a protocol is a rare event, but at large
> > scale we need this capability. Extending a protocol is far simpler
> > that replacing a protocol especially if we can avoid needing to swap
> > out massive amounts of hardware.
> >
> > Another general characteristic is that GUE extensions are contained
> > within a single header, as opposed to using shim headers or following
> > headers. There are benefits to do this:
> >
> > * It's very clear that the extensions refer to the encapsulation. For
> > instance if implement a checksum in a shim header then we need some
> > consideration as to how the checksum covers prior headers.
> > * We can limit the extensibility. This is important consideration for
> > efficiency and avoid DOS attacks.
> > * This allows "skip ahead". If a device just wants to look at the
> > encapsulated IP packet for instance it does not need to parse a header
> > chain for that.
> >
> > Thanks,
> > Tom
> >
> > _______________________________________________
> > nvo3 mailing list
> > [email-masked]
> > https://www.ietf.org/mailman/listinfo/nvo3
>
>
>
> _______________________________________________
> nvo3 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nvo3
>
>

---

From: Tom Herbert <[email-masked]>
Date: Fri, 19 Aug 2016 15:53:02 -0700

Hi Lizhong, thanks for the feedback! Some replies inline.

On Fri, Aug 19, 2016 at 8:07 AM, Lizhong Jin <[email-masked]> wrote:
> Thank
> [Lizhong] one more checksum will add additional load to hardware NAT, and it
> is required to update checksum for every NAT router which was not supposed
> to support GUE.
>
Right, the GUE checksum cannot be used through a NAT. In that case UDP
checksum is required. There's really no point in using UDP checksum
and GUE checksum simultaneously, the GUE checksum is really only for
those cases where a device is incapable of computing the full UDP
checksum over a packet (RX or TX).

> seems, to me, only outer or inner UDP checksum is enough.
I assume you mean inner transport checksum (e.g. TCP checksum) and
outer UDP checksum... Generally we found that it is better to set the
outer UDP checksum because 1) It provides coverage over more of the
packet including addresses which is especially important in IPv6 2) It
is better performance since deployed hardware, i.e. NICs, already
provide checksum offload for UDP and we are able to leverage that.

> [Lizhong] such kind of fragmentation is also not perfect. E.g., the receiver
> side hardware without reassembling could not get inner header, and fail to
> do RSS. I still like #3 as described in the draft.
>
RSS and ECMP are performed on outer IP addresses and UDP ports so we
don't lose that, these must be the same for all fragments of a packet.
DPI cannot be done on fragments (problem with any fragmentation
method), but it is also true for payload transform such as DTLS. One
of the goals of GUE (especially like in TOU) is to discourage and
eliminate DPI in middleboxes by encrypting the whole GUE payload.
Encryption is the only proposed solution to protocol ossification.

> also use label to implicitly identify a VPN, and it works well.
The GUE header security is primarily needed to verify the VNID. IMO in
a multi-tenant network the most critical requirement is to maintain
isolation between tenant networks. We need to protect against either
packets illegitimately crossing networks are someone attempting to
inject packets into a virtual network (the latter problem is
exacerbated by the use of UDP for encapsulations because that allows
non-priveleged applications to easily spoof packets to virtual
networks). The header security extension provides explicit security
for this. The payload transform option with DLTS should also be able
to secure the VNID as well. I would expect that we deploy with one of
those two options.

Thanks,
Tom

---

From: Lizhong Jin <[email-masked]>
Date: Sat, 20 Aug 2016 14:03:48 +0800

Hi Tom, see inline below.

Lizhong

> checksum over a packet (RX or TX).
[Lizhong] Then can I say, when checksum extension is deployed, the underlay
network
is restricted to be no NAT. This will be a limitation.

> provide checksum offload for UDP and we are able to leverage that.
[Lizhong] Since the link layer has already provide checksum, e.g., FCS in
Ethernet, it would be also safe to have only inner UDP checksum, right?
Anyway, if either inner or outer transport checksum is acceptable, does
that mean the motivation of RCO will not exist?

> receiver
[Lizhong] Relying on outer IP addresses and UDP ports maybe not a good way.
For the
IP fragmented packet, first&second fragments belong to same flow may have
different source
UDP ports, especially in the scenario where TS and NVE is separated. Method
of generating
the source UDP ports is not standardized, and using source UDP port may not
have even distribution of RSS hash value. E.g., 3bits of hash value maybe
enough for source UDP port, because usually ECMP in network will support 8
path. But RSS needs 7bits of hash value. Then the best way to do RSS for
fragments is to parse the inner header at receive side, and do the hash
based on 3/2-tuple as currently doing.