Subject: Re: [nvo3] Concerns about NVO3 dataplane requirements document

From: Erik Nordmark <[email-masked]>
Date: Tue, 04 Nov 2014 15:46:05 -0800

On 10/26/14 1:20 PM, Tom Herbert wrote:
> On Fri, Oct 24, 2014 at 5:44 PM, Erik Nordmark <[email-masked]> wrote:
>>
>> It would be good for the NVO3 WG to have a clear understanding of what data
>> needs to be carried with each encapsulated frame. That helps determine how
>> flexible and extensible the packet format needs to be.
>> The experience with extensibility for protocols that are in the dataplane
>> (be it IPv4 options, IPv6 extension headers, TRILL options, etc) is that
>> they don't tend to get implemented in hardware. And the dataplane protocols
>> tend to have a mixture of hardware and software implementations - which is
>> different than TCP which is mostly software.
> I don't believe this is always true. We have verified that at least
> two NICs and one switch chip are capable of parsing any combination of
> keyid, sequence number, and checksum fields in GRE for the purposes of
> calculating a flow hash on the inner header. In fact, we've been able
> to overload the sequence number and checksum fields for our own
> options in lieu of HW not supporting a general L3 extensibility
> mechanism (like IP options).
>

Tom,

I was referring to the case when the IETF defines some 
options/extensions mechanism. Those don't tend to get implemented in 
hardware.

Your example is where some existing hardware parser can be reused for 
some other purpose by overloading or redefining existing protocol 
fields. I've seen that elsewhere as well. But that doesn't refute the 
point about lack of hardware implementations for options/extensions.

Regards,
      Erik

---

From: Erik Nordmark <[email-masked]>
Date: Tue, 04 Nov 2014 15:54:07 -0800

On 10/25/14 11:28 PM, Marc Binderberger wrote:
> For my comment about IPSEC/AH:
>> One question is whether the higher assurance is just for the VNI or for the
>> whole encapsulated frame. Using something like ESP/AH takes us down the
>> path of protecting the whole frame, which might be overkill.
> agree. My (naive?) idea was to let AH in the NVO3 case only cover the
> underlay IP header, the NVO3 header and maybe the tenant frame header
> (IPv4/v6 or the Ethernet header). Honestly, have to re-read the specs if
> there is a way to chieve this. It would protect all the information needed
> when decapsulating the NVO3 overlay. The tenant frame protection I would
> leave to a separate AH/ESP step before the overlay encap and after decap.

Marc,

AFAIK all the use of AH/ESP assume that it protects to the end of the 
payload.
I guess one could define a variant - in the same way that UDP-lite is a 
variant of UDP - but it would require adding a "protected length" field 
somewhere.

>> As I tried to clarify in my response to Tom the meta-data discussion in the
>> IETF was mostly about vendor-specific service meta-data, but perhaps this
>> term is being used for more general extensibility?
> that's how I was thinking about it. This may work as long as the encap/decap
> entity - which can be a NIC, switch/router etc. - does not need to understand
> what the meta-data means and can just add/remove it as opaque data.

Depending on the semantics it might be hard to add data - e.g., if the 
data requires performing some computation on the packet.
The goal should be to at least be able to silently ignore received 
meta-data.

>> I think there should be ways to add better assurance (checksum, keyed
>> hashes) for the NVO3 header. But perhaps that can be in fixed fields in a
>> fixed length header.
> You mean the usual fields of "auth type", "key id" and such, typically part
> of an authentication TLV, should be part of the NVO3 header from the
> beginning? Nothing optional but build-in from the start?

Yes if the WG feels strongly about security it could define those fields 
in as fixed fields from day 1. Whether the fields are used would be a 
function of whether the security pieces are enabled - if not it would 
waste some bandwidth.

My point is that this is an option which results in a fixed header 
length with features that can be selectively enabled. Optional features 
doesn't have to imply variable header length.

>
>
>> In terms of the overall architecture there is a desire to carry some
>> service meta-data with frames. The sfc WG is thinking about doing that
>> using a separate NSH header.
> Even if we only cover IPv4, IPv6 and Ethernet as payload we need some
> protocol or next-header field, IMHO. This could cover e.g. NSH as well.
>
> Saying this, one could then also define a next-header value for OAM instead
> of having a OAM (aka RA) flag - and let the hardware trigger on the value
> instead of a flag.
> Nevertheless, I think a reserved field for flags is a good idea if we have
> the room in the NVO3 header.
>
>
Good point. If the OAM packets only need to be treated differently by 
the decapsulating node then a payload type would suffice.

I don't know whether some of the cases (now discussed in Lime) assume 
that a transit node can easily see which packets are OAM packets.

    Erik

---

From: Tom Herbert <[email-masked]>
Date: Tue, 4 Nov 2014 16:39:19 -0800

On Tue, Nov 4, 2014 at 3:46 PM, Erik Nordmark <[email-masked]> wrote:
> I was referring to the case when the IETF defines some options/extensions
> mechanism. Those don't tend to get implemented in hardware.
>
> Your example is where some existing hardware parser can be reused for some
> other purpose by overloading or redefining existing protocol fields. I've
> seen that elsewhere as well. But that doesn't refute the point about lack of
> hardware implementations for options/extensions.
>
I suspect there are several devices that implement keyid option in GRE
and nvgre (for VNID). In any case, the requirements document probably
needs to expound upon what the hardware requirements of the dataplane
are, especially if such requirements will potentially place
constraints on other requirements (like extensibility).

Thanks,
Tom

---

From: Erik Nordmark <[email-masked]>
Date: Thu, 06 Nov 2014 10:33:25 -0800

On 11/4/14 4:39 PM, Tom Herbert wrote:
> and nvgre (for VNID).
But the VNID isn't an optional extension so it is quite natural that 
hardware that implements nvgre would implement it.

The issue we see is with IETF standardizing some optional option; in 
general hardware implementations do not support those.

> In any case, the requirements document probably

I definitely don't think (re)using existing hardware should trump other 
requirements.

But it would be nice (desirable) if a new protocol (with good support 
for extensibility) can be implemented using existing silicon, even if 
that silicon can't handle the extensions themselves. That facilitates 
incremental deployment.

Regards,
    Erik

---

From: Tom Herbert <[email-masked]>
Date: Fri, 7 Nov 2014 09:18:12 -0800

On Thu, Nov 6, 2014 at 10:33 AM, Erik Nordmark <[email-masked]> wrote:
>>> some

Erik,

The VNID uses the keyid which is optional in the underlying
encapsulation which is GRE. When we deploy nvgre in the network we now
see two formats of GRE with different header lengths. Both of these
formats are already supported in switches for purposes of EMCP hash,
so we didn't need swap out any hardware to get "support" for nvgre. If
you want to say these are two different versions of a protocol, or
just two different protocols, as opposed to one version with and
without an optional extension-- that is fine, but semantically these
descriptions are equivalent.

> The issue we see is with IETF standardizing some optional option; in general
> hardware implementations do not support those.
>
> But it would be nice (desirable) if a new protocol (with good support for
> extensibility) can be implemented using existing silicon, even if that
> silicon can't handle the extensions themselves. That facilitates incremental
> deployment.
>
Agreed. I think nvgre/GRE demonstrates a possible direction for that.
Purposely limit the number of possible variants of the header, but
allow a means to (sparingly) add new variants. This works well in GRE
since number of fields that can be added is limited, and fields are
always well ordered (for instance with 5 bits, we'd have up to 32
header combinations which seems reasonable to put into a TCAM for
parsing).

A constrained extensibility approach precludes open-ended
extensibility that one might get with a rich set of TLVs and vendor
specific options. Proposals like NSH and geneve seem to allow for
that. This also should be considered in the requirements.

Tom

---

From: Marc Binderberger <[email-masked]>
Date: Mon, 10 Nov 2014 15:21:00 -0800

Hello Tom et al.,

hmm, looking at the original email from Erik I wonder: have we made any 
progress?

For the "extensions & hardware" discussion: there is obviously no simple 
truth. And ECMP is a well-defined task, this does not mean much for parsing 
general TLV. I understood Erik's comment as a "avoid TLVs if we can do it 
otherwise, e.g. with fixed+flag fields". 

Which is not far away from your idea to constrain, I think; nevertheless it 
requires to know today - at least roughly - what extensions you need. For the 
aspects where you know that you don't know yet I see two options:

(1) come up with a generic TLV scheme
(2) define a new header when you need it

Solution (1) is the obvious choice when options/TLVs are optional. When new 
fields are mandatory for the functionality then you may effectively end up 
with (2) - a "header plus mandatory TLV plus TLV sequence rules" is not 
different from a new header that looks like "old header plus TLV fields".

Reading through the various mails and drafts I come up with this first 
conclusion:

- VNI field of at least 24 bit is required (*)

- ECMP hashing is needed, either by an explicit field or by offloading it 
into the UDP source port value

- protocol field is needed. Personally I like to think of a "next-header" as 
this would allow to point to the payload protocol as well as to a TLV. Saves 
bits in the initial header design. The open question is 8 or 16 bit.

- version field. In case of an IP/UDP encapsulation I wonder if another UDP 
destination port would not do the same (?), again saving bits. 

- OAM. Proposals so far use a flag. With a protocol or next-header field I 
wonder if an "OAM" value could not do the same as the OAM flag. Actually it 
could do more as some may want a "punt, don't forward" OAM while others may 
have plans for a "punt & forward" OAM. This would be simply two different 
protocol/next-header values.

(*: Tom, you mentioned some VNI hierarchy but I don't think we came to a 
conclusion how many bits this would need and if the hierarchy requires 
dataplane support or can be handled by the control plane)

> extensibility) can be implemented using existing silicon, even if that 
silicon can't 
> handle the extensions themselves. That facilitates incremental deployment.

agree. Which practically means VxLAN or NVGRE (?). I would still prefer we 
think this from the requirements before mapping it onto one of the existing 
schemes. Not that I expect we come up with something fundamentally different 
(e.g. Geneve also looks "suspiciously" similar to VxLAN ;-) but it may help 
asking if fields are really required.

Regards, Marc

On Fri, 7 Nov 2014 09:18:12 -0800, Tom Herbert wrote:
>> general

---

From: "Chenhao (philips)" <[email-masked]>
Date: Tue, 11 Nov 2014 04:05:17 +0000

Hello all,

Please find comments in-line.

-hao
发件人: nvo3 [mailto:[email-masked]] 代表 Erik Nordmark
发送时间: 2014年10月21日 5:37
收件人: [email-masked]
主题: [nvo3] Concerns about NVO3 dataplane requirements document

I expressed this on the phone at the interim meeting and was asked to post with a  bit more detail.

According to the call the intended purpose of the requirements document is to help the WG choose between different proposed dataplane encapsulation protocols. However, I get the impression that draft-ietf-nvo3-dataplane-requirements was not written with that purpose in mind.

To be clear, the document contains useful background text and various discussion on what needs to be done at encapsulation and decapsulation points either by an implementation or operationally. However, that does not help with the (current) intended purpose.

The actual requirements on encapsulation protocol are few and weak. Section 3.3.1 contains a few of them.
But even the requirement on VNID is weak. The document says that there MUST be a VNID, but then goes on to

       This field MAY be an

       explicit, unique (to the administrative domain) virtual network

       identifier (VNID) or MAY express the necessary context information

       in other ways (e.g. a locally significant identifier).

While in theory locally significant indentifiers can be made to work, they would require an additional control-plane mechanism to handle the (dynamic?) mapping between VNID  and the local identifier. Especially with BGP EVPN now being handled in the BESS WG, I think we should have a requirement for our dataplane that it MUST contain a VNID field.

That section also discusses QoS/CoS. But it's requirement is a MAY for a QoS field in the NVO3 overlay header. Either we should require such a field or not; otherwise this doesn't help us choose.
(And my personal take is that we can have solutions which map between TS QoS/CoS and underlay CoS without also having some QoS field in the NVO3 header. But my overall point is that the "MAY" isn't a helpful criteria to choose between protocols.)

Those are the only "requirements" on the dataplane I've found in the document.

There might be other required or desired properties that are not in the draft. For instance, one can contemplate a requirement that the encapsulation MUST/SHOULD facilitate ECMP in unmodified routers in the underlay (e.g., using the common technique of UDP encaps with entropy placed in the UDP source port field).
[hao] Existing ECMP routing hashes in a per-flow fashion. All packets from a single flow will be delivered along the same path. It’s unreasonable for large flow which is very common in today’s datacenters. Hashing in a finer granularity rather than per-flow hashing will be helpful for load balancing. Actually there are already some work[FLARE][CONGA] which we may refer to.
  [FLARE] Kandula, S.; et al. “Dynamic Load Balancing Without Packet Reordering”, SIGCOMM Comput. Commun. Rev., ACM, 2007, 37, 51-62.
[CONGA] Alizadeh, M.; et al. “CONGA: Distributed Congestion-aware Load Balancing for Datacenters”, Conference on SIGCOMM, ACM, 2014, 503-514.

Thus my take is that a document with requirements on the NVO3 dataplane encapsulation can be a page or two. Some introduction and background followed by a list of required and desired properties of the  encapsulation format such as:
 - MUST contain an VNID field. This field MUST be large enough to scale to 100's of thousands of virtual networks
 - ??? QoS field inside the NVO3 overlay header or not ???
 - MUST/SHOULD facilitate ECMP in unmodified routers in the underlay
(Others participants might have other requirements on the encapsulation format. My main message is the focus on the encaps requirements which seems to be quite few.)

The implementation and operational requirements in draft-ietf-nvo3-dataplane-requirements should IMHO belong in a different document. Some might already be covered in the architecture document. And others might need to be refined as we specify more details for the NVO3 solution.

Hence if we are going to get WG consensus on a dataplane encapsulation requirements document I strongly suggest we start small with the above (3, give or take) bullets being the actual content.

   Erik

Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away.
Antoine de Saint-Exupery<http://www.quotationspage.com/quotes/Antoine_de_Saint-Exupery/>
French writer (1900 - 1944)

---

From: "Chenhao (philips)" <[email-masked]>
Date: Tue, 11 Nov 2014 04:21:33 +0000

Hello all,

-hao
发件人: nvo3 [mailto:[email-masked]] 代表 Erik Nordmark
发送时间: 2014年10月21日 23:37
收件人: [email-masked]<mailto:[email-masked]>
主题: [nvo3] Concerns about NVO3 dataplane requirements document

       This field MAY be an

   Erik

---

From: Erik Nordmark <[email-masked]>
Date: Thu, 13 Nov 2014 17:11:17 -1000

Marc,

One of the OAM needs I understand is the need to have OAM packets follow 
the same path as data packets, but not be forwarded to the TS after 
decap. Some OAM bit could have that semantics. Or some next-hdr="drop".

But I don't know what other OAM cases there are.

> agree. Which practically means VxLAN or NVGRE (?).

AFAIK the hardware is more flexible than that. But I haven't looked at 
the details.

    Erik

> I would still prefer we