Subject: [nvo3] FW: New Version Notification for draft-zhou-li-vxlan-soe-00.txt

From: "Zhou, Han" <[email-masked]>
Date: Fri, 14 Mar 2014 02:25:40 +0000

Hi folks,

We posted a draft as an extension to VXLAN. Please take a look.

The motivation came from our experiments on VXLAN optimization. It seems lots 
of discussions ongoing about the necessity of adding metadata to transport headers, 
and it is also controversial whether we should take offloading into consideration in
the headers. However, our test result shows significant performance gains even 
without any help from hardware offloading. The performance of a single TCP 
session improved from 1.5 Gbits/sec to 3.5~4 Gbits/sec: more than doubled!

So this is a practical yet generic proposal, which extends the offloading concept 
to from kernel stacks to remote end-points of overlay networks.

The metadata for offloading is very similar to STT. There difference is that:
1. it doesn’t add fake TCP header to utilize NIC TSO.
2. it doesn't include helper fields - just to save the limited VXLAN header space for 
other possible purpose in the future.
3. VXLAN is widely adopted and this is only a minor extension backward compatible

Based on this, it is highly recommended to add segmentation metadata in VXLAN 
header as proposed in this draft. 

Any comments are appreciated!

Best regards,
Han Zhou

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] 
Sent: Thursday, March 13, 2014 10:29 PM
To: Zhou, Han; Li, Chengyuan; Li, Chengyuan; Zhou, Han
Subject: New Version Notification for draft-zhou-li-vxlan-soe-00.txt

A new version of I-D, draft-zhou-li-vxlan-soe-00.txt
has been successfully submitted by Han Zhou and posted to the
IETF repository.

Name:		draft-zhou-li-vxlan-soe
Revision:	00
Title:		Segmentation Offloading Extension for VxLAN
Document date:	2014-03-13
Group:		Individual Submission
Pages:		7
URL:            http://www.ietf.org/internet-drafts/draft-zhou-li-vxlan-soe-00.txt
Status:         https://datatracker.ietf.org/doc/draft-zhou-li-vxlan-soe/
Htmlized:       http://tools.ietf.org/html/draft-zhou-li-vxlan-soe-00

Abstract:
   Segmentation offloading is nowadays common in network stack
   implementation and well supported by para-virtualized network device
   drivers for virtual machine (VM)s. This draft describes an extension
   to Virtual eXtensible Local Area Network (VXLAN) so that segmentation
   can be decoupled from physical/underlay networks and offloaded
   further to the remote end-point thus improving data-plane performance
   for VMs running on top of overlay networks.

Please note that it may take a couple of minutes from the time of submission
until the htmlized version and diff are available at tools.ietf.org.

The IETF Secretariat

---

From: Tom Herbert <[email-masked]>
Date: Fri, 14 Mar 2014 08:13:01 -0700

> session improved from 1.5 Gbits/sec to 3.5~4 Gbits/sec: more than doubled!
Hi Han,

- The mechanisms you're using are local within a host so this should
be accomplished by software API as opposed to changing the on-the-wire
protocol. The API should be generic other encaps. In the most general
case, we'd want to provide a TSO device interface to the guest and
only do the segmentation at last possible point in the stack (either
GSO from the host's physical driver, or TSO if device has support).
Most of this is supported now in Linux kernel.
- I believe this would conflict with the proposal to add a protocol
field to the VXLAN header. Overloading one field in a fixed header is
not an adequate substitute for a truly extensible header. In the best
case we could only use one or the other functionality in a given
packet. In the worse case, overloading opens the door to backwards
compatibility issues and the potential for misinterpretation of
fields.

Tom

> _______________________________________________
> nvo3 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nvo3

---

From: "Zhou, Han" <[email-masked]>
Date: Sat, 15 Mar 2014 03:52:56 +0000

Hi Tom,

Thanks for your comments! Please see my comments inline.

> From: Tom Herbert [mailto:[email-masked]]
> Sent: Friday, March 14, 2014 11:13 PM
> To: Zhou, Han
> Cc: [email-masked]; [email-masked]
> Subject: Re: [nvo3] FW: New Version Notification for
> draft-zhou-li-vxlan-soe-00.txt
> 

Tom, the point of this draft is that the "last possible point in the stack" can
be pushed to the remote end-point of the VXLAN tunnel. If the remote
is an hypervisor, this GSO is terminated without actual work: the 
receiving hypervisor simply delivers the large packet to receiving guest.
This results in a huge performance gain, especially in the receiving side
because the number of packets being handled are much smaller. The 
speed up is 2x - 3x even when the physical network still transmits small
(1514 bytes) packets with IP fragmentation. If jumbo frames used in
physical network the speedup will be boosted even higher. I would like 
to share more data of the prototype if it is of interest.

So this is not a local mechanism, but need agreement between end-points.
In a setup like:
   Hypervisor A <-- Hypervisor B --> Gateway
On hypervisor B the VTEP treat both remote VTEPs the same way: it fills
segmentation-offloading information for GSO packets. Hypervisor A
optionally checks such information to understand that this is a valid large
packet and don't drop it even its size is bigger than the guest's virtual
interface. But this information is critical to the Gateway: it has to perform
the real segmentation if the packet is being forwarded to physical networks.
And this is why we need the metadata in the on-the-wire protocol.

> compatibility issues and the potential for misinterpretation of

I agree that we'd better avoid field overloading. But as stated in section 3,
it is not conflict with VXLAN-gpe, because when segmentation offloading
is enabled the encapsulated packet should always be Ethernet, and in 
such case prototype is not needed. But you reminded me that, it should 
be defined clearly that P bit specified by VXLAN-gpe MUST be set to 0 
when S bit is 1. We will address that in version 01.

Or if you know any real (or potential) scenarios of conflict between 
VXLAN-soe and VXLAN-gpe, please kindly point out and we can consider
using the remaining space in the header instead of overloading it.

> > URL:
> http://www.ietf.org/internet-drafts/draft-zhou-li-vxlan-soe-00.txt

---

From: Tom Herbert <[email-masked]>
Date: Sat, 15 Mar 2014 11:06:17 -0700

> using the remaining space in the header instead of overloading it.
If I were using VXLAN then I'd most likely use VXLAN-gpe and the
common case would be protocol=0x800 (IPv4). I'm sure I'd want the full
benefits of any offload capability, so it seems like using your
feature in that case isn't feasible because of the field overloading.

Adding a bit for each interpretation of a fixed field illustrates
another problem. Each use of the field must be mutually exclusively of
the other uses. In this case the possible values for P and S bit are
00, 10, 01-- however 11 is nonsensical. If someone adds another
interpretation of the field (related to OAM bit maybe), then this
results in four legitimate values (000 100 010 001) in the three bits
and four nonsensical values (110 011 101 111). For a fourth
interpretation, there's 5 good values and so 11 bad values, and so on.
It is an inefficient use of bit fields to design in invalid
combinations of bits. I would suggest instead that three or four bits
be grouped together as a type field. For example, in three bits 7
unique uses of the fixed field can be expressed.

Tom

---

From: "Zhou, Han" <[email-masked]>
Date: Mon, 17 Mar 2014 08:13:50 +0000

Hi Tom,

I agree with you that these two extensions to VXLAN (GPE and SOE) 
should co-exist and they are not mutually exclusive. At least in below 
scenario:

VM1 <-> HV1 <=> GW1 <=> GW2 <=> HV2 <-> VM2

HV (Hypervisor) <=> GW (Gateway): VXLAN encap L2
GW1 <=> GW2: VXLAN encap L3

For GW1 <=> GW2, GPE is required to encap L3, and SOE is also 
preferred to push Segmentation Offloading to the remote GW.
VM1 send large packets to HV1 (via TSO supported by virtual driver), 
and HV1 doesn't perform the segmentation but push offloading to
GW1 (via VXLAN-soe) => GW2 => HV2, and finally HV2 delivers the 
large packet to VM2 without performing any segmentation, thus the 
performance gains achieved.

So I would consider below change to the VXLAN-soe header to separate
"overlay MTU" to high and low bytes:
    0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |S|C|R|R|I|R|R|R| OverlayMTU HI | <Protocol Type (used by GPE)> |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |           VXLAN Network Identifier (VNI)      | OverlayMTU LO |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

This would avoid conflicting with VXLAN-gpe. And it would be even better if 
VXLAN-gpe could consider shrink the protocol type field if only a limited 
number of ethertypes such as IPv4 and IPv6 are required. That would save
much space in VXLAN header for other purpose. Copy VXLAN-gpe authors.

Best regards,
Han Zhou

> Sent: Sunday, March 16, 2014 2:06 AM
> To: Zhou, Han
> Cc: [email-masked]; [email-masked];
> [email-masked]
> >> > 2. it doesn't include helper fields - just to save the limited VXLAN header
> space
> >> for

---

From: Tom Herbert <[email-masked]>
Date: Mon, 17 Mar 2014 19:28:23 -0700

> "overlay MTU" to high and low bytes:
IP protocol can be used instead, this can cover ipv4,6 and Ethernet. That's
a savings of 8 bits.

The MTU probably could be a scaled value (>>4 for instance) so something
less than 16 bits might work.

I think you can come up with something functional but I'm not sure it's
something you'll be really proud of. Another consideration is whether this
functionality is so important and well understood that you'd want to commit
precious space to it in the header and not reserve the space for some
critical need later.... I don't know how to answer that question.

>> >> > So this is a practical yet generic proposal, which extends the
offloading
>> concept
>> >> > 3. VXLAN is widely adopted and this is only a minor extension
backward
>> >> compatible
>> >> >
>> >> > Based on this, it is highly recommended to add segmentation
metadata in
>> >> VXLAN
>> >> > https://www.ietf.org/mailman/listinfo/nvo

---

From: Erik Nordmark <[email-masked]>
Date: Mon, 24 Mar 2014 11:04:01 -0700

> receiving hypervisor simply delivers the large packet to receiving guest.
Han,

Are you saying that the sending VM and vswitch will send a large packet 
(e.g., 32k) over UDP, and this will be delivered to the receiving 
vSwitch and VM as one large packet? That certainly makes the work on the 
VMs a lot less, hence I can understand that you see performance 
improvements.

However, that would result in IP fragmentation of that large UDP/VXLAN 
packet AFAICT.

The IETF has some experience with protocols where the loss unit is 
smaller than the retransmission unit, and this results in very poor 
performance under packet loss due to the loss of a single, small unit 
resulting in the retransmission of a large unit (the 32k packet, which 
might be a TCP, SCTP, etc packet i.e. a reliable protocol with 
retransmissions.)

Regards,
     Erik

---

From: Tom Herbert <[email-masked]>
Date: Mon, 24 Mar 2014 14:33:55 -0700

On Mon, Mar 24, 2014 at 10:34 AM, Erik Nordmark <[email-masked]> wrote:
> (e.g., 32k) over UDP, and this will be delivered to the receiving vSwitch
> and VM as one large packet? That certainly makes the work on the VMs a lot
> less, hence I can understand that you see performance improvements.
>
That would be UFO. It seems more likely that you'd want to do TSO/LRO
style L4 segmentation/reassembly.

It's still not clear to me on why the MTU would need to be on the
wire, offload mechanisms already work now without that information.
Also, for packets going from one VM to another within a host moving
jumbo packets could be directly linked into the TSO/LRO mechanisms.

> The IETF has some experience with protocols where the loss unit is smaller
> than the retransmission unit, and this results in very poor performance
> under packet loss due to the loss of a single, small unit resulting in the
> retransmission of a large unit (the 32k packet, which might be a TCP, SCTP,
> etc packet i.e. a reliable protocol with retransmissions.)
>
> Regards,
>     Erik
>
>
>
>
>>>> offloading concept
>>>> 2. it doesn't include helper fields - just to save the limited VXLAN
>>>> header space
>>>
>>> for
>>>>
>>>> 3. VXLAN is widely adopted and this is only a minor extension backward
>>>
>>> compatible
>>>>
>>>> Based on this, it is highly recommended to add segmentation metadata in
>>>
>>> VXLAN
>>>>
>>>> Status:
>>>> https://datatracker.ietf.org/doc/draft-zhou-li-vxlan-soe/

---

From: "Zhou, Han" <[email-masked]>
Date: Tue, 25 Mar 2014 02:02:07 +0000

Hi Erik,

Thanks for pointing out. Yes, your understanding is correct.
 
However, the issue might not be so serious considering that physical
networks within data-center are usually highly reliable, which is the
main usecase for this protocol.

Implementation can choose to make this feature configurable, and 
enable only under reliable environment. 

I will add this consideration into version 01.

Best regards,
Han

---

From: "Zhou, Han" <[email-masked]>
Date: Tue, 25 Mar 2014 02:23:43 +0000

Hi Tom,

> style L4 segmentation/reassembly.

UFO would be ideal, but it is not available in most current NICs. And it
is not the mechanism described in this draft. This proposal is a pure
software solution without any hardware offloading required.
Implementation can choose to utilize hardware UFO if available, and
otherwise fallback to the mechanism proposed here (VXLAN-SOE).

> jumbo packets could be directly linked into the TSO/LRO mechanisms.

If the remote VTEP is a gateway rather than a hypervisor, it would
require the overlay MTU information for re-segmentation so that the
correct MTU (may be a result of path-MTU negotiation) is applied. As
mentioned in the draft (section 2.4), there are 3 choices on the gateway:
1. re-segment in gateway software
2. offload to in NIC hardware
3. offload to the next hop if it is a tunnel transport (e.g. VXLAN-GPE)
and the tunnel protocol supports offloading, too (e.g. also VXLAN-SOE).

Best regards,
Han