Subject: [nvo3] FW: I-D Action: draft-zhou-li-vxlan-soe-01.txt

From: "Zhou, Han" <[email-masked]>
Date: Tue, 20 May 2014 03:01:42 +0000

Hi,

We have updated the VXLAN-SOE draft according to earlier comments. Now it is fully compatible with VXLAN-GPE. And some examples are added for better understanding.

A prototype is also implemented here (patch based on Open vSwitch):
https://github.com/hzhou8/openvswitch/commit/9a7deb8b432ce83a9c09d7d4ff85fa050f7dd2be

netperf TCP_STREAM test result between a pairs of VMs on hosts with 10G interfaces:

Before the change: 2.62 Gbits/sec
After the change: 6.68 Gbits/sec
Speedup is ~250%.

The patch attracted some interests in OVS community, but since this RFC draft is in very early stage so it is regarded as inappropriate by Jesse to apply the change to OVS tree.
The discuss mail-thread: 
http://openvswitch.org/pipermail/discuss/2014-May/013981.html
http://openvswitch.org/pipermail/discuss/2014-May/013898.html

So we would like to request a review here by NVO3/TOFOO groups and VXLAN authors: is this VXLAN extension is worth formally put into VXLAN as a standard, so that more people can benefit from it?

Best regards,
Han

-----Original Message-----
From: I-D-Announce [mailto:[email-masked]] On Behalf Of [email-masked]
Sent: Friday, May 02, 2014 8:09 PM
To: [email-masked]
Subject: I-D Action: draft-zhou-li-vxlan-soe-01.txt

A New Internet-Draft is available from the on-line Internet-Drafts directories.

        Title           : Segmentation Offloading Extension for VXLAN
        Authors         : Han Zhou
                          Chengyuan Li
	Filename        : draft-zhou-li-vxlan-soe-01.txt
	Pages           : 13
	Date            : 2014-05-02

Abstract:
   Segmentation offloading is nowadays common in network stack
   implementation and well supported by para-virtualized network device
   drivers for virtual machine (VM)s. This draft describes an extension
   to Virtual eXtensible Local Area Network (VXLAN) so that segmentation
   can be decoupled from physical/underlay networks and offloaded
   further to the remote end-point thus improving data-plane performance
   for VMs running on top of overlay networks.

The IETF datatracker status page for this draft is:
https://datatracker.ietf.org/doc/draft-zhou-li-vxlan-soe/

There's also a htmlized version available at:
http://tools.ietf.org/html/draft-zhou-li-vxlan-soe-01

A diff from the previous version is available at:
http://www.ietf.org/rfcdiff?url2=draft-zhou-li-vxlan-soe-01

Please note that it may take a couple of minutes from the time of submission
until the htmlized version and diff are available at tools.ietf.org.

Internet-Drafts are also available by anonymous FTP at:
ftp://ftp.ietf.org/internet-drafts/

_______________________________________________
I-D-Announce mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/i-d-announce
Internet-Draft directories: http://www.ietf.org/shadow.html
or ftp://ftp.ietf.org/ietf/1shadow-sites.txt

---

From: Tom Herbert <[email-masked]>
Date: Tue, 20 May 2014 11:43:58 -0700

Hi Zou, a couple of questions inline.

On Mon, May 19, 2014 at 8:01 PM, Zhou, Han <[email-masked]> wrote:

> Hi,
>
> We have updated the VXLAN-SOE draft according to earlier comments. Now it
> is fully compatible with VXLAN-GPE. And some examples are added for better
> understanding

A prototype is also implemented here (patch based on Open vSwitch):
> Can you provide some more details on this benefit? It seems like plain
TSO/LRO that understands encapsulation should provide similar benefits when
going between hosts.

The patch attracted some interests in OVS community, but since this RFC
> draft is in very early stage so it is regarded as inappropriate by Jesse to
> apply the change to OVS tree.
> The discuss mail-thread:
> So we would like to request a review here by NVO3/TOFOO groups and VXLAN
> authors: is this VXLAN extension is worth formally put into VXLAN as a
> standard, so that more people can benefit from it?
>
> Could you get the same effect by setting larger MTUs on the overlay
network interface and relying in path MTU discovery when going over
physical network?

> Internet-Draft<https://www.ietf.org/mailman/listinfo/i-d-announceInternet-Draft>directories:
> http://www.ietf.org/shadow.html

---

From: "Zhou, Han" <[email-masked]>
Date: Wed, 21 May 2014 00:28:39 +0000

Hi Tom,

Thanks for your comments.
Yes TSO/LRO with VXLAN support should provide similar or even better performance gains, but the mechanism proposed by this draft decouples overlay and underlay, and it is hardware independent.
Secondly, hardware offloading usually support TCP only (TSO). The mechanism here can help on large UDP packet performance, also verified by the prototype.
BTW, how many types of off-the-shelf NIC support VXLAN offloading? Any performance data?

Likewise, setting large MTU on overlay interfaces achieves similar result, but still the overlay/underlay decoupling issue. It is usually advised that overlay MTU is slightly smaller than underlay to avoid inefficient fragmentation after adding the outer header, but to achieve really high performance between VMs, large MTU is preferred. And considering overlay <-> physical connection, path MTU discovery is not always work.
This kind of configuration complexity and pain-point can be resolved simply by decoupling overlay and underlay MTU, as suggested by this draft. Here is an example of configuration confusion:
http://openvswitch.org/pipermail/discuss/2014-May/013898.html

Ideally, all NV tunnel protocols should support similar metadata, thus overlay segmentation can be offloaded hop-by-hop.

Best regards,
Han

From: Tom Herbert [mailto:[email-masked]]
Sent: Wednesday, May 21, 2014 2:44 AM
To: Zhou, Han
Cc: [email-masked]; [email-masked]; [email-masked]; [email-masked]; Erik Nordmark
Subject: Re: FW: I-D Action: draft-zhou-li-vxlan-soe-01.txt

On Mon, May 19, 2014 at 8:01 PM, Zhou, Han <[email-masked]<mailto:[email-masked]>> wrote:
Hi,

We have updated the VXLAN-SOE draft according to earlier comments. Now it is fully compatible with VXLAN-GPE. And some examples are added for better understanding

Before the change: 2.62 Gbits/sec
After the change: 6.68 Gbits/sec
Speedup is ~250%.
Can you provide some more details on this benefit? It seems like plain TSO/LRO that understands encapsulation should provide similar benefits when going between hosts.

So we would like to request a review here by NVO3/TOFOO groups and VXLAN authors: is this VXLAN extension is worth formally put into VXLAN as a standard, so that more people can benefit from it?
Could you get the same effect by setting larger MTUs on the overlay network interface and relying in path MTU discovery when going over physical network?

Best regards,
Han

-----Original Message-----
From: I-D-Announce [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of [email-masked]<mailto:[email-masked]>
Sent: Friday, May 02, 2014 8:09 PM
To: [email-masked]<mailto:[email-masked]>
Subject: I-D Action: draft-zhou-li-vxlan-soe-01.txt

Abstract:

Please note that it may take a couple of minutes from the time of submission
until the htmlized version and diff are available at tools.ietf.org<http://tools.ietf.org>.

[email-masked]<mailto:[email-masked]>

---

From: "Zhou, Han" <[email-masked]>
Date: Wed, 21 May 2014 01:01:27 +0000

Hi Joe,

Thanks for your comment. 

Yes you are right that "length" fields in packet headers can be regarded as hard limit of MTU, but here in the draft we were referring interface MTU. We will make it more precise in next versions.

For your question, it seems there are misunderstandings. It is always Guest OS (VM) handles the TCP, but segmentation is offloaded to host. Let me explain the code change:
- TX side:
-- before the change:
  TCP segmentation offloaded by TSO of Guest OS virt-driver from guest to host, MSS carried by GSO metadata in skbuff.
  VXLAN layer add encapsulation, and overlay TCP segmentation is carried right before sending to host IP layer, which is the idea of GSO - segment at the latest point.
  Host do IP fragmentation only if overlay segment + outer header exceed physical interface MTU. (e.g. when both guest MTU and host MTU are configured to 1500)
-- after the change:
  TCP segmentation offloaded by TSO of Guest OS virt-driver from guest to host, MSS carried by GSO metadata in skbuff.
  VXLAN layer removes GSO metadata and store it in VXLAN-SOE header. So overlay TCP segmentation is skipped.
  Host do IP fragmentation according to physical interface MTU.

-RX side:
-- before the change:
  Host do IP reassembly if necessary. (e.g. when both guest MTU and host MTU are configured to 1500)
  Overlay TCP segments are decapsulated and delivered all the way to guest, and guest OS do TCP handling. (high cost here)
-- after the change:
  Host do IP reassembly, and large packets decapsulated and delivered to guest, and guest OS do TCP handling. (cost reduced here because of reduced number of packets)

I hope this clarifies.

Best regards,
Han

> From: Joe Touch [mailto:[email-masked]]
> Sent: Wednesday, May 21, 2014 1:29 AM
> To: Zhou, Han; [email-masked]; [email-masked];
> [email-masked];
> [email-masked]
> Cc: Erik Nordmark; Tom Herbert
> Subject: Re: [Tofoo] FW: I-D Action: draft-zhou-li-vxlan-soe-01.txt
> 
> Hi, all,
> 
> I had a comment and a question:
> 
> Comment - (from the doc) overlays do have a hard MTU limit; it is the
> limit of the encapsulation mechanism. E.g., without additional layers,
> for UDP in IPv4 this would be a at most 65507 bytes (i.e., IPv4 max -
> (min IP header + UDP header)). Using additional adaptation layers, this
> could be larger (e.g., see SEAL).
> 
> Question - the code appears to have the VXLAN layer do the
> fragmentation, with the OS layer implementing the rest of TCP. There are
> a lot of interactions, notably:
> 
> 	- any mechanism outside of the TCP source and TCP destination
> 	that interprets the TCP header will result in a decrease in
> 	functionality
> 		i.e., the TCP connection will support only the
> 		intersection of options and features supported
> 		by the source, dest, *and* VXLAN layers
> 
> 		(rather than being limited only by the
> 		source-dest pair)
> 
> 	- if passed a full TCP segment, this mechanism will be
> 	incompatible with TCP security (e.g., TCP MD5, TCP-AO, and
> 	the results of the TCPCRYPT WG.
> 
> I'm not quite sure from your doc whether you're re-segmenting TCP
> segments, or merely collecting them for aggregate transit (e.g., as is
> done in burst-mode Ethernet).
> 
> Can you please clarify?
> 
> Joe
> 
> 
> On 5/19/2014 8:01 PM, Zhou, Han wrote:
> > Hi,
> >
> understanding.
> > The patch attracted some interests in OVS community, but since this RFC draft
> is in very early stage so it is regarded as inappropriate by Jesse to apply the
> change to OVS tree.
> > The discuss mail-thread:
> authors: is this VXLAN extension is worth formally put into VXLAN as a standard,
> so that more people can benefit from it?
> >
> > Best regards,
> > Han
> >
> > Tofoo mailing list
> > [email-masked]
> > https://www.ietf.org/mailman/listinfo/tofoo
> >

---

From: "Zhou, Han" <[email-masked]>
Date: Wed, 21 May 2014 02:17:24 +0000

Hi Joe,

This is an interesting topic.

> TCP offloading is fine when the OS hands off user data, and the offload
> engine creates the entire segment.
Existing TSO/GSO mechanisms deliver full (large) TCP segment to "offload engine", which then create smaller segments according to physical MTU, and recalculates checksums.
This is the case even without overlay considered. So I suppose the problem you pointed out is not related to my change, but a general limitation for TSO/GSO, right?

For my understanding the TCP implementation should decide whether to use offloading or not according to the feature/options required by a TCP connection. If the option required (such as MD5) is not supported by offloading, the TCP stack should do the segmentation by itself instead of utilizing offloading.

In fact, the proposal in this draft should be able to alleviate the limitation for TCP connections between VMs behind same gateways, because in this case there is no real TCP segmentation performed by "offload engine". 

Let me know if you have more concerns, or maybe an example of how an option is broken by TSO/GSO, then we can check what's the current solution in kernel.

Best regards,
Han

> Sent: Wednesday, May 21, 2014 9:14 AM
> Hi, Han,
> This helps, but doesn't quite address my concern.
> 
> TCP offloading is fine when the OS hands off user data, and the offload
> engine creates the entire segment.
> 
> The situation you're describing seems to be a hybrid, where the guest OS
> makes a TCP segment, and then something lower (in the VM system) parses
> that segment to create one or more segments on the wire.
> 
> If that happens, it will be incompatible with a number of existing TCP
> options, and will also cause side-effects with ACK clocking, timeouts,
> and more than a few other TCP features.
> 
> Although I appreciate the goal of efficiency and speed, there is a
> severe compatibility issue that doesn't seem to be addressed.
> 
> We can discuss this off-list if useful, FWIW.
> 
> Joe
> 
> On 5/20/2014 6:01 PM, Zhou, Han wrote:
> > Hi Joe,
> >
> > Thanks for your comment.
> >
> > Yes you are right that "length" fields in packet headers can be regarded as hard
> limit of MTU, but here in the draft we were referring interface MTU. We will
> make it more precise in next versions.
> >
> > For your question, it seems there are misunderstandings. It is always Guest OS
> (VM) handles the TCP, but segmentation is offloaded to host. Let me explain the
> code change:
> > - TX side:
> > -- before the change:
> >    TCP segmentation offloaded by TSO of Guest OS virt-driver from guest to
> host, MSS carried by GSO metadata in skbuff.
> >    VXLAN layer add encapsulation, and overlay TCP segmentation is carried
> right before sending to host IP layer, which is the idea of GSO - segment at the
> latest point.
> >    Host do IP fragmentation only if overlay segment + outer header exceed
> physical interface MTU. (e.g. when both guest MTU and host MTU are configured
> to 1500)
> > -- after the change:
> >    TCP segmentation offloaded by TSO of Guest OS virt-driver from guest to
> host, MSS carried by GSO metadata in skbuff.
> >    Overlay TCP segments are decapsulated and delivered all the way to guest,
> and guest OS do TCP handling. (high cost here)
> > -- after the change:
> >    Host do IP reassembly, and large packets decapsulated and delivered to
> guest, and guest OS do TCP handling. (cost reduced here because of reduced
> number of packets)
> >
> > I hope this clarifies.
> >
> > Best regards,
> > Han
> >
> draft

---

From: Tom Herbert <[email-masked]>
Date: Tue, 20 May 2014 19:42:28 -0700

On Tue, May 20, 2014 at 5:28 PM, Zhou, Han <[email-masked]> wrote:

> Yes TSO/LRO with VXLAN support should provide similar or even better
> performance gains, but the mechanism proposed by this draft decouples
> overlay and underlay, and it is hardware independent.
>
> Secondly, hardware offloading usually support TCP only (TSO). The
> mechanism here can help on large UDP packet performance, also verified by
> the prototype.
>
 BTW, how many types of off-the-shelf NIC support VXLAN offloading? Any
> performance data?
HW is not a requirement for this offloading. There has been a lot of work
recently to get software variants (GSO and GRO) working in Linux with
tunnels. These do show 2-3x performance improvements. HW support would be
mostly an incremental improvement to that, or becomes interesting for OS
bypass like in SR-IOV. GSO/GRO can be presented to the guest as TSO and LRO
so that it's possible to plumb use of large packets from the guest all the
way to the host driver. It should also be possible to plumb two VMs on the
same host to communicate without segmentation, i.e. output from TSO on one
VM becomes input for another.

The advantage that I see in your draft is that it allows an intermediate
device to perform segmentation/reassembly instead of fragmentation.

 Likewise, setting large MTU on overlay interfaces achieves similar result,
> but still the overlay/underlay decoupling issue. It is usually advised that
> overlay MTU is slightly smaller than underlay to avoid inefficient
> fragmentation after adding the outer header, but to achieve really high
> performance between VMs, large MTU is preferred.
>

Depends on the performance dimension to be optimized. Larger MTUs could
increase latency of high priority small packets for instance (HOL
blocking), or UDP based application might try to use MTU to decide how
large it should send it's packets to avoid fragmentation.

And considering overlay <-> physical connection, path MTU discovery is not
> always work.
 This kind of configuration complexity and pain-point can be resolved
> simply by decoupling overlay and underlay MTU, as suggested by this draft.
> Here is an example of configuration confusion:
>
> Ideally, all NV tunnel protocols should support similar metadata, thus
> overlay segmentation can be offloaded hop-by-hop.
>

I think this would be applicable to about all tunnel protocols. Then you
would also want to do reassembly at each hop? Sounds expensive. Once you
segment, I think you'd only only want to reassemble at the end host.

One important thing to keep in mind, and a hard lesson in real deployment
:-), segmentation offload is opportunistic and very dependent on the
conditions of the traffic. It's value can be fleeting in real workloads.
For instance imagine a host with a lot of active high throughout
connections (like a video serving) which hits a hiccup causing all
congestions windows to shrink. If the system is not provisioned for this
event it can be very hard to recover (a lot more CPU is required to achieve
same throughput). This differentiates a larger MTU from relying on
segmentation, the former offers more predictable CPU savings.

Tom

>
> Best regards,
>
> Han
>
>
>
> *From:* Tom Herbert [mailto:[email-masked]]
> *Sent:* Wednesday, May 21, 2014 2:44 AM
> *To:* Zhou, Han
> *Cc:* [email-masked]; [email-masked];
> [email-masked]; Erik Nordmark
> *Subject:* Re: FW: I-D Action: draft-zhou-li-vxlan-soe-01.txt
>
>
>

---

From: "Zhou, Han" <[email-masked]>
Date: Wed, 21 May 2014 03:07:11 +0000

Hi Tom,

> There has been a lot of work recently to get software variants (GSO and GRO) working in Linux with tunnels.
Are you referring UDP GRO for VXLAN? I am aware of this implementation in kernel 3.14. The performance gain should be similar, but still requires careful configuration of VM MTU to avoid IP fragmentation on physical MTU, rather than decoupling from underlay. I think each of these solutions has its own advantages.

> I think this would be applicable to about all tunnel protocols. Then you would also want to do reassembly at each hop? Sounds expensive. Once you segment, I think you'd only only want to reassemble at the end host.
There is no segmentation/reassembly needed at each hop, but instead “offload” hop by hop, just save the metadata to the next hop headers. And finally segmentation should be avoided if the destination is a VM on a hypervisor.
This scenario is like in Example 2 (section 5.2), just replace the VXLAN-SOE between the 2 gateways by any other Tunnel protocol that can support similar offloading feature, such as STT.

Best regards,
Han

From: Tom Herbert [mailto:[email-masked]]
Sent: Wednesday, May 21, 2014 10:42 AM
To: Zhou, Han
Cc: [email-masked]; [email-masked]; [email-masked]; [email-masked]; Erik Nordmark
Subject: Re: FW: I-D Action: draft-zhou-li-vxlan-soe-01.txt

On Tue, May 20, 2014 at 5:28 PM, Zhou, Han <[email-masked]<mailto:[email-masked]>> wrote:
Hi Tom,

HW is not a requirement for this offloading. There has been a lot of work recently to get software variants (GSO and GRO) working in Linux with tunnels. These do show 2-3x performance improvements. HW support would be mostly an incremental improvement to that, or becomes interesting for OS bypass like in SR-IOV. GSO/GRO can be presented to the guest as TSO and LRO so that it's possible to plumb use of large packets from the guest all the way to the host driver. It should also be possible to plumb two VMs on the same host to communicate without segmentation, i.e. output from TSO on one VM becomes input for another.

The advantage that I see in your draft is that it allows an intermediate device to perform segmentation/reassembly instead of fragmentation.

Likewise, setting large MTU on overlay interfaces achieves similar result, but still the overlay/underlay decoupling issue. It is usually advised that overlay MTU is slightly smaller than underlay to avoid inefficient fragmentation after adding the outer header, but to achieve really high performance between VMs, large MTU is preferred.

Depends on the performance dimension to be optimized. Larger MTUs could increase latency of high priority small packets for instance (HOL blocking), or UDP based application might try to use MTU to decide how large it should send it's packets to avoid fragmentation.

And considering overlay <-> physical connection, path MTU discovery is not always work.
This kind of configuration complexity and pain-point can be resolved simply by decoupling overlay and underlay MTU, as suggested by this draft. Here is an example of configuration confusion:
http://openvswitch.org/pipermail/discuss/2014-May/013898.html

I think this would be applicable to about all tunnel protocols. Then you would also want to do reassembly at each hop? Sounds expensive. Once you segment, I think you'd only only want to reassemble at the end host.

One important thing to keep in mind, and a hard lesson in real deployment :-), segmentation offload is opportunistic and very dependent on the conditions of the traffic. It's value can be fleeting in real workloads. For instance imagine a host with a lot of active high throughout connections (like a video serving) which hits a hiccup causing all congestions windows to shrink. If the system is not provisioned for this event it can be very hard to recover (a lot more CPU is required to achieve same throughput). This differentiates a larger MTU from relying on segmentation, the former offers more predictable CPU savings.

Tom

Best regards,
Han

From: Tom Herbert [mailto:[email-masked]<mailto:[email-masked]>]
Sent: Wednesday, May 21, 2014 2:44 AM
To: Zhou, Han
Cc: [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; Erik Nordmark
Subject: Re: FW: I-D Action: draft-zhou-li-vxlan-soe-01.txt

Best regards,
Han

-----Original Message-----
From: I-D-Announce [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of [email-masked]<mailto:[email-masked]>
Sent: Friday, May 02, 2014 8:09 PM
To: [email-masked]<mailto:[email-masked]>
Subject: I-D Action: draft-zhou-li-vxlan-soe-01.txt

Abstract:

---

From: Tom Herbert <[email-masked]>
Date: Wed, 21 May 2014 11:29:30 -0700

On Tue, May 20, 2014 at 8:07 PM, Zhou, Han <[email-masked]> wrote:

> Are you referring UDP GRO for VXLAN? I am aware of this implementation in
> kernel 3.14.
>

GRO and GSO for UDP tunnels, VXLAN is one use case.

> The performance gain should be similar, but still requires careful
> configuration of VM MTU to avoid IP fragmentation on physical MTU, rather
> than decoupling from underlay. I think each of these solutions has its own
> advantages.
>
>
In the case of TCP it's really the path MTU that is relevant. We really
want the MSS to be the largest value which avoids fragmentation in the path.

For UDP it is less straightforward. If the VM MTU is greater than the
physical MTU or even the path MTU then fragmentation is possible. Most UDP
applications really want to avoid fragmentation (much worse than doing TCP
segmentation), so they need to ascertain what size to send packets-- either
they choose a fixed minimum value, use the MTU, or some out of band
negotiation. With the occurrence of transport protocols running over UDP
(such as QUIC), this becomes important. In this case, it's really hard to
meaningfully decouple physical MTU (path MTU) from those of the overlay.

>
>
> > I think this would be applicable to about all tunnel protocols. Then
> you would also want to do reassembly at each hop? Sounds expensive. Once
> you segment, I think you'd only only want to reassemble at the end host.
>
> There is no segmentation/reassembly needed at each hop, but instead
> “offload” hop by hop, just save the metadata to the next hop headers. And
> finally segmentation should be avoided if the destination is a VM on a
> hypervisor.
>
> This scenario is like in Example 2 (section 5.2), just replace the
> VXLAN-SOE between the 2 gateways by any other Tunnel protocol that can
> support similar offloading feature, such as STT.
>
>
>
This where your proposal confuses me, we already have a lot of offloading
support of protocols without having to change protocols to support
offloading. I don't know if the concept of offloading is generic enough to
need intrinsic protocol support, or if it is at what protocol level this
should be (for instance I could conceive of this being an option in TCP to
allow mega sized segments).

A example where use case SOE is usable but GSO/GRO wouldn't be might be
enlightening!

Thanks,
Tom

 Best regards,
> *Sent:* Wednesday, May 21, 2014 10:42 AM
>
> *To:* Zhou, Han
>  HW is not a requirement for this offloading. There has been a lot of
> work recently to get software variants (GSO and GRO) working in Linux with
>  Likewise, setting large MTU on overlay interfaces achieves similar
> result, but still the overlay/underlay decoupling issue. It is usually
> advised that overlay MTU is slightly smaller than underlay to avoid
> inefficient fragmentation after adding the outer header, but to achieve
> really high performance between VMs, large MTU is preferred.
>
>
>

---

From: Joe Touch <[email-masked]>
Date: Tue, 20 May 2014 10:28:41 -0700

Hi, all,

Can you please clarify?

Joe

---

From: Joe Touch <[email-masked]>
Date: Tue, 20 May 2014 18:14:28 -0700

Hi, Han,

Joe

---

From: Joe Touch <[email-masked]>
Date: Wed, 21 May 2014 10:02:02 -0700

On 5/20/2014 7:17 PM, Zhou, Han wrote:
> Existing TSO/GSO mechanisms deliver full (large) TCP segment to
> "offload engine", which then create smaller segments according to
> physical MTU, and recalculates checksums. This is the case even
> without overlay considered. So I suppose the problem you pointed out
> is not related to my change, but a general limitation for TSO/GSO,
> right?

It depends on what part of TCP happens in the guest OS vs. the 
underlying engine. If you expose the TCP API to the guest OS (the API 
spec'd in RFC793), and hand "Send" call data down to the engine, that's 
fine.

However, what I think is happening is this:

	- the guest OS receives the "Send" call and creates a TCP
	segment, including TCP header and TCP options

	- the guest OS hands the TCP segment to the engine

	- the engine parses that TCP segment to create multiple
	outgoing segments, typically by copying the passed segment's
	header and options, and recalculating the fields it
	thinks it needs to

Simply put, that's as bad as having any middlebox re-calculating TCP 
segments, and is guaranteed to create problems (even if the 'typical' 
case doesn't trip over them).

The problem is that the engine's TCP interpreter may not understand all 
TCP header options - when (not if) that happens, what does it do?

RFC793 is clear on this - when a SYN arrives with an option that isn't 
understood, the receiver MUST silently ignore that option.

So the engine ought to have stripped out all options it doesn't 
understand from the first SYN sent*. But I suspect that's not what it 
thinks it should do - I suspect it thinks it's OK to merely copy - or 
pass through - options it doesn't understand.

What should happen is that the engine interface should NEVER be a TCP 
segment formed by the guest OS. If what you want is to offload 
segmentation, you ought to pass the user data and TCP header (and its 
options) as separate parameters.

(* this is why a correctly-written engine ends up reducing TCP 
functionality, because a connection can support only what is supported 
by the endpoints AND the engine [on each end]). Any option the engine 
doesn't support should never be allowed on the connection.

> For my understanding the TCP implementation should decide whether to
> use offloading or not according to the feature/options required by a TCP
> connection. If the option required (such as MD5) is not supported by
> offloading, the TCP stack should do the segmentation by itself instead
> of utilizing offloading.

That works if unknown options are assumed NOT SUPPORTED.

But I still don't quite understand why you want the segmentation 
happening in the VM - why not pass the MTU info to the virtual interface 
in the guest OS and let it handle things?

> In fact, the proposal in this draft should be able to alleviate the
> limitation for TCP connections between VMs behind same gateways, because
> in this case there is no real TCP segmentation performed by "offload
> engine".
>
> Let me know if you have more concerns, or maybe an example of how an
> option is broken by TSO/GSO, then we can check what's the current
> solution in kernel.

See above - and thanks,

Joe

---

From: "Zhou, Han" <[email-masked]>
Date: Thu, 22 May 2014 02:44:52 +0000

Hi Joe,

Thanks for your explain, and in theory I tend to agree with you.
However, look at the real world implementation of offload, e.g. Linux kernel, GSO is implemented in the way I described: a whole TCP packet (including headers and options) are generated by the TCP layer, and segmentation is performed as late as possible, depending on the FEATURES supported by net-device:
1) TSO is NOT supported by net-device then GSO software segmentation is performed (tcp_gso_segment())
2) TSO is supported by NIC, then segmentation is offloaded to NIC hardware. (in this case tcp_dump will not capture small packets but only the large packets before segmentation)
3) It is in Guest OS and TSO is supported by the virtual net-device, then segmentation is offloaded to host OS. 

And my change is specific for 3): without the change segmentation is performed according to the MSS specified by guest OS,  and with the change this segmentation is skipped.

As you can see the compatibility problem you mentioned, if it is a real problem, is not introduced by my change.

Because of this, we might discuss it somewhere else, such as linux kernel community.
I don't think current Linux kernel GSO implementation addresses the problem you mentioned (I could be wrong). Do you know any example of *correct* implementation of TCP offloading? Or it could be a TODO in linux kernel.

Best regards,
Han
> Sent: Thursday, May 22, 2014 1:02 AM

---

From: Joe Touch <[email-masked]>
Date: Thu, 22 May 2014 11:16:35 -0700

> As you can see the compatibility problem you mentioned, if it is a real problem, is not introduced by my change.

Perhaps, but dealing with guest OS/TSO interactions could be designed as 
a correct example, rather than replicating the failure of other 
implementers.

> I don't think current Linux kernel GSO implementation addresses the problem you mentioned (I could be wrong). Do you know any example of *correct* implementation of TCP offloading? Or it could be a…

I haven't studied TSO implementations; there are too many, and many are 
proprietary.

Joe

---

From: Tom Herbert <[email-masked]>
Date: Thu, 22 May 2014 11:43:41 -0700

On Tue, May 20, 2014 at 6:14 PM, Joe Touch <[email-masked]> wrote:

> This would essentially be TOE.

> This how Linux and probably about every NIC implements TCP segmentation
offload. A large TCP segment is broken up into MSS sized segments (MSS is
provided in ancillary data). The process replicates the IP/TCP headers per
packet, adjusts the length, sequence numbers, and computes checksum for
each segment.  There's no special handling of any options, it is assumed
they can be replicated in each segment.

> options, and will also cause side-effects with ACK clocking, timeouts, and
> more than a few other TCP features.

Although I appreciate the goal of efficiency and speed, there is a severe
> compatibility issue that doesn't seem to be addressed.
>
> TCP segmentation offload is already widely deployed. Since this draft
would be using the same mechanism it's unlikely to create new compatibility
issues except for the fact that the segmentation might be deferred to an
off host entity which is a new concept that would probably have side
effects.

Tom

>> Yes you are right that "length" fields in packet headers can be regarded
>> as hard limit of MTU, but here in the draft we were referring interface
>> MTU. We will make it more precise in next versions.
>>
>> For your question, it seems there are misunderstandings. It is always
>> Guest OS (VM) handles the TCP, but segmentation is offloaded to host. Let
>> me explain the code change:
>> - TX side:
>> -- before the change:
>>    TCP segmentation offloaded by TSO of Guest OS virt-driver from guest
>> to host, MSS carried by GSO metadata in skbuff.
>> right before sending to host IP layer, which is the idea of GSO - segment
>> at the latest point.
>> physical interface MTU. (e.g. when both guest MTU and host MTU are
>> configured to 1500)
>> -- after the change:
>>    TCP segmentation offloaded by TSO of Guest OS virt-driver from guest
>> to host, MSS carried by GSO metadata in skbuff.
>>    Overlay TCP segments are decapsulated and delivered all the way to
>> guest, and guest OS do TCP handling. (high cost here)
>> -- after the change:
>>>> netperf TCP_STREAM test result between a pairs of VMs on hosts with 10G
>>>>
>>> interfaces:
>>>
>>>>
>>>> From: I-D-Announce [mailto:[email-masked]] On Behalf Of
>>>>
>>> [email-masked]
>>>

---

From: Joe Touch <[email-masked]>
Date: Thu, 22 May 2014 11:50:04 -0700

On 5/22/2014 11:43 AM, Tom Herbert wrote:
>
>
>
> On Tue, May 20, 2014 at 6:14 PM, Joe Touch <[email-masked]
> <mailto:[email-masked]>> wrote:
>
>     Hi, Han,
>
>     TCP offloading is fine when the OS hands off user data, and the
>     offload engine creates the entire segment.
>
>     The situation you're describing seems to be a hybrid, where the
>     guest OS makes a TCP segment, and then something lower (in the VM
>     system) parses that segment to create one or more segments on the wire.
>
> offload. A large TCP segment is broken up into MSS sized segments (MSS
> is provided in ancillary data). The process replicates the IP/TCP
> headers per packet, adjusts the length, sequence numbers, and computes
> checksum for each segment.  There's no special handling of any options,
> it is assumed they can be replicated in each segment.

That's an incorrect assumption.

>     If that happens, it will be incompatible with a number of existing
>     TCP options, and will also cause side-effects with ACK clocking,
>     timeouts, and more than a few other TCP features.
>
>
> would be using the same mechanism it's unlikely to create new
> compatibility issues except for the fact that the segmentation might be
> deferred to an off host entity which is a new concept that would
> probably have side effects.

IMO, this draft should not rely on or further promote a mechanism with 
known problems.

Joe

---

From: Tom Herbert <[email-masked]>
Date: Thu, 22 May 2014 14:02:48 -0700

On Thu, May 22, 2014 at 11:50 AM, Joe Touch <[email-masked]> wrote:

> If there are options that can't be replicated then the stack won't use the
offload mechanism for those cases. Of course in that situation the
performance benefits would be lost, so we really like new options and
fields in protocols to be able to be replicated across a few segments :-).
New instances of per packet checksums, lengths, sequence numbers, etc. make
segmentation offload harder.

> The mechanism is not inherently problematic, it is still up to the
implementation to ensure correctness. What is different with SOE is that
the offload functionality becomes visible in the protocol and creates cross
implementation dependencies to implement correctly.

Tom

---

From: Joe Touch <[email-masked]>
Date: Thu, 22 May 2014 14:10:46 -0700

On 5/22/2014 2:02 PM, Tom Herbert wrote:
...
> If there are options that can't be replicated then the stack won't use
> the offload mechanism for those cases. Of course in that situation the
> fields in protocols to be able to be replicated across a few segments
> :-). New instances of per packet checksums, lengths, sequence numbers,
> etc. make segmentation offload harder.

Segmentation should never need to happen in an offload engine unless 
that's where headers are generated. It's never appropriate to merely 
repeat TCP options - consider that this affects timestamps, ACKs, etc.

>     IMO, this draft should not rely on or further promote a mechanism
>     with known problems.
>
> The mechanism is not inherently problematic,it is still up to the
> the offload functionality becomes visible in the protocol and creates
> cross implementation dependencies to implement correctly.

I don't agree about the lack of inherent problems.

Joe