Subject: [nvo3] HW offload requirements (was Re: Consensus call on encap proposals)

From: Tom Herbert <[email-masked]>
Date: Tue, 9 Aug 2016 07:33:41 -0700

On Mon, Aug 8, 2016 at 12:43 AM, Lizhong Jin <[email-masked]> wrote:
> Hi,
> With my design experience of NIC and Switch, I prefer VXLAN-GPE. I have the
> same concern of HW complicated logic from Fabio, and additional concern to
> GENEVE and GUE is its long size of header.
> 1. GENEVE: 256+8=262Bytes
> 2. GUE: 128+4=132Bytes

This doesn't seem to take into account that NSH provides extensibility
for VXLAN-GPE as Fabio mentioned. If you were to take this into
account how does that affect HW complexity argument?

> In many current switch and NIC design, the parser buffer size is still
> 128Bytes, and some NIC/DMA has 256Bytes buffer. This is workable because:
> 1. IPv4 options is not widely used.
> 2. IPv6 extension header only support with limited number.

The limit of IPv6 extension headers is currently the maximum MTU of
the packet. Having no "small" limit is a problem for some hardware
that want to parse past them. There was a long discussion about this
in v6ops. One proposed idea to limit IPv6 extension headers to occupy
at most N bytes (N based on the parse buffer as you described). But
there was no agreement on what N should be, and it was pointed out
that not all implementations even have this problem and there is no
reason to believe HW won't continue to improve to make such an
artificial limit obsolete.

> But if adding GENEVE/GUE header, the minimum size of buffer is 256Bytes, or
> even 512Bytes. Then the question is, does the hardware need to process these
> Variable Length Options/Optional Fields/Private Data. If not, then it is not
> reasonable to force the hardware to increase the cost, but gain little.
>
Can you clarify this? If hardware does not need to process variable
length options then wouldn't that mean it does not need to support
them? This is in fact is the case with the many of NICs and switches
in deployment don't have protocol specific support for these
encapsulations. For such legacy HW we can leverage support for UDP
offloads (ECMP, RSS, checksum offload) to get really good solutions.
Hardware offloads of encapsulation protocols can be good optimization
(personally I am hoping we can offload DTLS for GUE), but I don't see
how HW support is a requirement nor how we are forcing hardware to
increase cost.

Thanks,
Tom

> Lizhong
>
>
>
>>
>> ---------- Forwarded message ----------
>> From: Alia Atlas <[email-masked]>
>> To: NVO3 <[email-masked]>
>> Cc: "Bocci, Matthew (Nokia - GB)" <[email-masked]>
>> Date: Fri, 29 Jul 2016 11:13:00 -0400
>> Subject: Re: [nvo3] Consensus call on encap proposals
>> I'd like to have people focus on the key point of this thread.
>>
>> Are there serious technical objections (and specifically what are they) to
>> moving forward with VXLAN-GPE as the standards-track protocol?
>>
>> Are there serious technical objections (and specifically what are they) to
>> moving forward with GENEVE as the standards-track protocol?
>>
>> Are there serious technical objections (and specifically what are they) to
>> moving forward with GUE as the standards-track protocol?
>>
>> We need to capture any relevant objections.  So far, there's been some
>> discussion on extensibility - with Tom Herbert providing concrete concerns.
>>
>> I have concluded that almost all the authors would prefer to have no
>> standards track solution if they can't guarantee that theirs is that
>> standard.
>>
>> I do hear concerns about whether a decision will be too late.   I think
>> that a decision can only be helpful.   It goes back to when is the best time
>> to plant a tree - with the answer of 20 years ago or now.
>>
>> Regards,
>> Alia
>>
>>
>> On Fri, Jul 29, 2016 at 4:34 AM, Naoki Matsuhira
>> <[email-masked]> wrote:
>>>
>>>
>>>
>>> On 2016/07/21 23:56, Bocci, Matthew (Nokia - GB) wrote:
>>>>
>>>> WG
>>>>
>>>> There was a discussion in the NVO3 WG meeting in Berlin following strong
>>>> advice from our Area Director that we need to come to a consensus on
>>>> converging on a common encapsulation. Two sets of questions were asked:
>>>> (1) Should the WG move forward with one standards track encap?
>>>> (2) For a given encap, do you have significant technical objections?
>>>
>>>
>>> I want to inform to this mailing list that I proposed ME6E-FP and ME6E-PR
>>> at the yokohama meeting. I also have proposal M46E-FP and M46E-PR (past
>>> called SA46T).
>>>
>>> These encapsulation technologies are based on address mapping. ME6E use
>>> IPv6 address which mapping MAC address, and M46E use IPv6 address which
>>> mapping IPv4 address.
>>>
>>> I understand too many encapsulation technologies, however these my
>>> proposal are simple, and may contribute to the Internet.
>>>
>>> I believe address mapping approach is unique, so I want to propose again.
>>>
>>> sorry not the answer to the question.
>>>
>>> Naoki Matsuhira
>>>
>>>
>>> _______________________________________________
>>> nvo3 mailing list
>>> [email-masked]
>>> https://www.ietf.org/mailman/listinfo/nvo3
>>
>>
>>
>
>
> _______________________________________________
> nvo3 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nvo3
>

---

From: Lizhong Jin <[email-masked]>
Date: Wed, 10 Aug 2016 09:10:22 +0800

Regards

Lizhong

On 08/09/2016 22:33, Tom Herbert wrote:

Lizhong: yes, if consider NSH as Fabio suggest, I need to rethink.

increase cost.

Lizhong: hardware will not recognize the variable length opinion, but it needs to skip these options, and to get inner packet header to do offloading, e.g., RSS as you mentioned. Then hardware surely needs to buffer the long header including variable length options. And we need to increase the parser buffer.

Thanks,

Tom

---

From: Jesse Gross <[email-masked]>
Date: Thu, 11 Aug 2016 21:53:03 +0000

> On Aug 9, 2016, at 6:10 PM, Lizhong Jin <[email-masked]> wrote:

Lizhong,

NICs that can offload Geneve already exist today - these are production models from major vendors. They will perform checksum offload, TSO, RSS, etc. in the presence of options. To give a concrete example, you can buy an Intel XL710, plug it into a Linux server, configure options using Open vSwitch and see offloading. Again, all of this is released hardware and software and something you can try yourself. The same thing is also possible with other recent NICs and platforms.

In general, the NICs that I’m aware of with this functionality can support 64 bytes of options. This is less than the maximum available in Geneve but still quite a bit of extensibility. While I expect that this will be sufficient for the foreseeable future, I don’t think that it means that we need to limit the maximum length in the protocol to the current support in hardware today - if there is a significant use for more options in the future then NIC buffers can always be increased as necessary. Better to have flexibility in the protocol and not need it than to not have the choice.

Jesse

---

From: Lizhong Jin <[email-masked]>
Date: Fri, 12 Aug 2016 11:21:55 +0800

Jesse, see inline below.

On Fri, Aug 12, 2016 at 5:53 AM, Jesse Gross <[email-masked]> wrote:

> > Lizhong: hardware will not recognize the variable length opinion, but it
> needs to skip these options, and to get inner packet header to do
> offloading, e.g., RSS as you mentioned. Then hardware surely needs to
> buffer the long header including variable length options. And we need to
> increase the parser buffer.
>
> Lizhong,
>
> NICs that can offload Geneve already exist today - these are production
> models from major vendors. They will perform checksum offload, TSO, RSS,
> etc. in the presence of options. To give a concrete example, you can buy an
> Intel XL710, plug it into a Linux server, configure options using Open
> vSwitch and see offloading. Again, all of this is released hardware and
> software and something you can try yourself. The same thing is also
> possible with other recent NICs and platforms.
>
> In general, the NICs that I’m aware of with this functionality can support
> 64 bytes of options. This is less than the maximum available in Geneve but
> still quite a bit of extensibility. While I expect that this will be
> sufficient for the foreseeable future, I don’t think that it means that we
> need to limit the maximum length in the protocol to the current support in
> hardware today - if there is a significant use for more options in the
> future then NIC buffers can always be increased as necessary. Better to
> have flexibility in the protocol and not need it than to not have the
> choice.
>
[Lizhong] I could not comment on specific vendor's product. And yes, some
architecture could achieve very long parser header, like software do. But
some architecture could not, mainly because of the delay.
To be clear, I am not opposing the long parer header, but concern of long
header while most options are not related with hardware. E.g., most routers
will implement IPv6 hop-by-hop option because HW needs to handle it. But as
I see, hardware does not need to handle Geneve options, but has to skip
them to get inner header.

Regards
Lizhong