Subject: [Spud] Segment offload for UDP-based protocols Re: [QUIC] Requirements

From: Brian Trammell <[email-masked]>
Date: Thu, 9 Jun 2016 12:41:29 +0200

hi Randall,

> On 09 Jun 2016, at 12:30, Randall Stewart <[email-masked]> wrote:
> 
> 
>> On Jun 9, 2016, at 5:06 AM, Michael Tuexen <[email-masked]> wrote:
>> 
>> 
>>> On 09 Jun 2016, at 10:03, Scharf, Michael (Nokia - DE) <[email-masked]> wrote:
>>> 
>>> My concern is that the current charter suggestion markets the protocol by “minimizing (…) overall transport latency for applications”. IMHO this can have rebound effects.
>>> 
>>> Here is what I mean by wrong expectations: Whether a protocol is indeed fast or not depends on many parameters including aspects such as hardware acceleration/offload (which matters at 10G/40G line speed, to the best of my knowledge). Some of these effects of
>> I think this refers to high throughput. For TCP, offloading TCP processing to a NIC improves performance
>> of the TCP processing. I'm not sure if this still helps if encryption/decryption comes into the game.
> 
> I too was thinking Michael meant offloading such as TSO and LRO. I know on most of the caches
> I develop on if I turn off TSO and LRO performance of the box drops by roughly 50%… and I think
> this will be a consequence of encrypting everything since I don’t think you can do TSO/LRO unless
> the hardware/lower-layers has access to things equivalent to TCP sequence numbers and ACK values,
> which is what I believe the encryption is meant to hide (from middle boxes).
> 
> That will be an extreme price to pay the question is, is it worth it /and or/ is there some other way
> to get the benefits of the TSO/LRO even with encryption… :-)

(adding [email-masked], as this is a relevant question for PLUS, too):

The question is what is the minimum information xSO/LRO needs to work. If I understand how generic segment offload works, if QUIC(/PLUS) exposes:

(1) a header field that can be used to order packets and find gaps that the kernel can find and interpret,

(2) a defined and easily-found boundary between header and payload, such that the payload retains any framing/message boundaries needed to meet API contracts, and

(3) rules about which header to choose for the "big" segment on receive

then adding QUIC(/PLUS) aware SO/LRO to the kernel should be reasonably simple.  I don't see why you need ACK-equivalents to be exposed to make this work, but I might be missing something.

Hardware offload takes longer to deploy, obviously, but runs on equivalent information.

Cheers,

Brian

---

From: Randall Stewart <[email-masked]>
Date: Thu, 9 Jun 2016 07:02:52 -0400

> On Jun 9, 2016, at 6:41 AM, Brian Trammell <[email-masked]> wrote:
> 
> hi Randall,
> 

My understanding of TSO (which is where we find the most benefit since its hardware) is the hardware gets a *large* segment (say 64k) and
cuts it up into MTU sized pieces duplicating the IP header and doing the right things with the tcp-sequence number.

If the sequence number is encrypted and not exposed that would be rather difficult with Quic (or any such protocol for
that matter even TCP if it was wrapped in UDP and encrypted).

The other benefit is of course comes from LRO and thats really not done in hardware but in software (at least in the
implementation on which I work that is) .. where Acks <or> Segments are combined into bigger chunks. 

For my work the Acks are what I care about having to process less Acks is a  win (but not nearly as much as TSO). 
Both Seq and Ack can be compiled by LRO into a “larger segment” so you need again the exposure of the Ack value as well
as the sequence number..

R

> _______________________________________________
> QUIC mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/quic

--------
Randall Stewart
[email-masked]
803-317-4952

---

From: Ian Swett <[email-masked]>
Date: Tue, 21 Jun 2016 15:16:53 -0400

I would agree with you Brian that QUIC/PLUS could be offloaded if #2 was
present.  #3 is not an issue if the header is encrypted, since it's just
part of the encrypted segment.  If it wasn't encrypted, I would suggest it
either be omitted from the non-first packets or identical across all
packets.  But either way, it's a solvable problem.

That being said, I think we should think hard about whether we want to
solve that problem, because it involves a lot of complexity and I believe
we'd be better served in ensuring there are simple send/receive paths that
are very fast.

In regards to acks and LRO, some middleboxes/endpoints will drop all but
the most recent ack if multiple arrive in close proximity.  It's not clear
this is always a feature from a congestion control perspective, but it
likely increases throughput in some environments.  However, given QUIC/PLUS
can easily change the receiver's behavior, we can send fewer acks.  Ideally
the sender would be aware of that, so it can adjust it's congestion control
to expect long pauses.  I think this achieves the same goal end-to-end and
CPU benchmarks in QUIC indicate it performs very well.

I'm hugely concerned about the CPU performance of QUIC, but nothing has
convinced me LRO or TSO would help enough to be worth the work.  Instead,
sending fewer acks and faster send/receive APIs show a lot more promise, at
least on the host side.

On Thu, Jun 9, 2016 at 6:41 AM, Brian Trammell <[email-masked]> wrote:

> >>> My concern is that the current charter suggestion markets the protocol
> by “minimizing (…) overall transport latency for applications”. IMHO this
> can have rebound effects.
> >>>
> >>> Here is what I mean by wrong expectations: Whether a protocol is
> indeed fast or not depends on many parameters including aspects such as
> hardware acceleration/offload (which matters at 10G/40G line speed, to the
> best of my knowledge). Some of these effects of
> >> of the TCP processing. I'm not sure if this still helps if
> encryption/decryption comes into the game.
> >
> The question is what is the minimum information xSO/LRO needs to work. If
> I understand how generic segment offload works, if QUIC(/PLUS) exposes:
>
> (2) a defined and easily-found boundary between header and payload, such
> that the payload retains any framing/message boundaries needed to meet API
> contracts, and
>
> then adding QUIC(/PLUS) aware SO/LRO to the kernel should be reasonably
> simple.  I don't see why you need ACK-equivalents to be exposed to make
> this work, but I might be missing something.
>