Subject: [iccrg] Fwd: New Version Notification for draft-welzl-irtf-iccrg-tcp-in-udp-00.txt

From: Michael Welzl <[email-masked]>
Date: Wed, 23 Mar 2016 09:02:38 +0100

Chair hat on:

We just put a presentation of the draft below on the agenda:
https://datatracker.ietf.org/meeting/95/agenda/iccrg/
...which still has space. Note that the duration written next to presentation reflects the presenters' wishes, not our ultimate decision, so there may be more time available than it seems.

If you have something to present, this is the right time to tell us! Don't wait until the last minute please.

Chair hat off:

Dear all,

I'm submitting this to ICCRG as the primary (and possibly ultimate? We'll see) forum to discuss this draft. We'd be happy to get your comments - please keep them on the ICCRG list.
We have FreeBSD code, from which we can already confirm that the encapsulation is indeed relatively easy to do (as our draft states). However, it doesn't do the congestion control coupling yet; we're working on it. For that, so far, we have simulations - from these, we hope to be able to show some first results in BA.

Cheers,
Michael

> Begin forwarded message:
> 
> From: <[email-masked]>
> Subject: New Version Notification for draft-welzl-irtf-iccrg-tcp-in-udp-00.txt
> Date: 21 Mar 2016 18:45:57 CET
> To: Jianjie You <[email-masked]>, Michael Welzl <[email-masked]>, Safiqul Islam <[email-masked]>, Kristian Hiorth <[email-masked]>
> Resent-From: <[email-masked]>
> 
> 
> A new version of I-D, draft-welzl-irtf-iccrg-tcp-in-udp-00.txt
> has been successfully submitted by Michael Welzl and posted to the
> IETF repository.
> 
> Name:		draft-welzl-irtf-iccrg-tcp-in-udp
> Revision:	00
> Title:		TCP in UDP
> Document date:	2016-03-21
> Group:		Individual Submission
> Pages:		17
> URL:            https://www.ietf.org/internet-drafts/draft-welzl-irtf-iccrg-tcp-in-udp-00.txt
> Status:         https://datatracker.ietf.org/doc/draft-welzl-irtf-iccrg-tcp-in-udp/
> Htmlized:       https://tools.ietf.org/html/draft-welzl-irtf-iccrg-tcp-in-udp-00
> 
> 
> Abstract:
>   This document specifies a method to encapsulate multiple TCP
>   connections using only one UDP port number pair.  Doing so allows for
>   a relatively easy implementation of coupled congestion control for
>   the TCP connections.  This can have several performance benefits, and
>   it makes it possible to precisely assign a share of the congestion
>   window to the connections based on priorities.  It also enables use
>   of UDP-based NAT traversal techniques, and it can act as a framework
>   for experimentation with novel changes to the TCP standard.
> 
> 
> 
> 
> 
> Please note that it may take a couple of minutes from the time of submission
> until the htmlized version and diff are available at tools.ietf.org.
> 
> The IETF Secretariat
>

---

From: Michael Welzl <[email-masked]>
Date: Fri, 25 Mar 2016 09:19:59 +0100

Hi all,

My apologies, I made a mistake: first I said that this should be discussed here in ICCRG, then I answered a comment on a different list and we ended up having a congestion control related discussion there.
Just what I wanted to avoid - I’d really like to collect the discussion of this draft in this list.

This email is me trying to undo this mistake and set the context for continuing the discussion here. The dialogue happened on the IAB stackevo-discuss list, and you can find it in its archives here:
https://mailarchive.ietf.org/arch/search/?email_list=stackevo-discuss

To summarize:
- Tom Herbert suggested that, instead of UDP encapsulation, one could just use the IPv6 flow label. That seems very reasonable to me and unless someone debates that, we’ll incorporate this in the next version of the draft (for the v6 case).
- Joe Touch raised issues about the idea of such tunneling, and pointed to an INTAREA tunnels draft for an explanation of these issues. I got surprised by this  but I’m not an expert on tunneling, so I’ll read this and try to understand the problem there.
- Joe Touch had comments related to congestion control; his point of view is that the UDP header is unnecessary because you could couple congestion controls even when packets traverse different paths. Here I disagree. My next email will continue this discussion with him, here on ICCRG.

Cheers,
Michael

> On 23. mar. 2016, at 09.02, Michael Welzl <[email-masked]> wrote:
> 
> Chair hat on:
> 
> _______________________________________________
> iccrg mailing list
> [email-masked]
> https://www.irtf.org/mailman/listinfo/iccrg

---

From: Tom Herbert <[email-masked]>
Date: Fri, 25 Mar 2016 10:00:33 -0700

Hi Michael,

For encapsulation format I suggest that you look at GUE
(https://tools.ietf.org/html/draft-ietf-nvo3-gue-02). GUE can
encapsulate TCP in UDP and inserts a GUE header that allows extra meta
data.  The connection ID could be moved to the GUE header so that no
special modification of the encapsulated TCP header would be needed.
The connection ID would be a new field in the GUE header. There is
already a session ID option defined, but that is 96 bits which is
probably overkill.

Another consideration is the UDP checksum. This must be set in IPV6
(excepting that the requirements in RFC6935 and RFC6936 are met) and
is recommended for encapsulation any way. This means that there are
two checksums (TCP and UDP) per packet which becomes a performance
issue since most NICs can offload at most one checksum and often that
is restricted to only the checksum in a plain TCP or UDP packet with
no encapsulation. We solved this in GUE with remote checksum offload
(https://tools.ietf.org/html/draft-herbert-remotecsumoffload-02) which
should work fine with TCP over UDP case.

There are some other end system issues in dealing with UDP
encapsulation that we considered in Linux (load balancing, checksum,
segmentation offload). Please look at
http://people.netfilter.org/pablo/netdev0.1/papers/UDP-Encapsulation-in-Linux.pdf.

Thanks,
Tom

On Wed, Mar 23, 2016 at 1:04 AM, Michael Welzl <[email-masked]> wrote:
> Hi,
>
> This could be of interest to folks here. We'd be glad to get your comments - please send them to the ICCRG list only, at [email-masked].
>
> Cheers,
> Michael
>
>
>> Begin forwarded message:
>>
>> From: Michael Welzl <[email-masked]>
>> Subject: Fwd: New Version Notification for draft-welzl-irtf-iccrg-tcp-in-udp-00.txt
>> Date: 23 Mar 2016 09:02:38 CET
>> To: [email-masked]
>>
>> (..)
>>
>> Chair hat off:
>>
>> Dear all,
>>

---

From: Michael Welzl <[email-masked]>
Date: Fri, 25 Mar 2016 18:29:00 +0100

Hi!

> On 25. mar. 2016, at 18.00, Tom Herbert <[email-masked]> wrote:
> 
> Hi Michael,
> 

I have, as I wrote this draft. It struck me as something that isn’t really necessary just for the purpose of what TCP-in-UDP is trying to achieve. In particular, other than GUE (if I got it correctly), TCP-in-UDP strictly assumes end-to-end operation, and requires changing the sender-side TCP code too because the main point is to combine congestion controls. This makes it possible to do whatever we want with the encapsulated TCP header.

> should work fine with TCP over UDP case.

We (and the preceding proposals that are cited in the draft) removed the redundant TCP checksum from the encapsulated TCP header. Because it’s end-to-end and changes TCP anyway, we can do that.

> http://people.netfilter.org/pablo/netdev0.1/papers/UDP-Encapsulation-in-Linux.pdf.

Thanks for the pointer!  But from quickly looking at it, the TCP-in-UDP case really seems simpler than most of this. E.g., in fig.1, a host kernel has to handle packets coming from a guest kernel - it’s not changing the guest kernel’s TCP code.

BTW I’m not at all insisting on my encapsulation method, just explaining the design rationale. Anyway I’m beginning to think that the next version of the draft should make a cleaner split between the two elements: 1) congestion control coupling and 2) encapsulation (ensuring that TCP packets take the same path). The latter could be a separate method to support 2), where various options could be described: for v6, using the flow label as you suggested; for v4, the existing UDP encapsulation or something else if there’s a good reason (first I’ll need to read some more encapsulation stuff more carefully to fully appreciate the reasons).

Cheers,
Michael

---

From: Michael Welzl <[email-masked]>
Date: Fri, 25 Mar 2016 19:22:08 +0100

> On 25. mar. 2016, at 18.58, Tom Herbert <[email-masked]> wrote:
> 
> On Fri, Mar 25, 2016 at 10:29 AM, Michael Welzl <[email-masked]> wrote:
>> Hi!
>> 
>> 
> Then you are redefining TCP.

Yes - that’s the whole point

> You can do that, but this substantially
> reduces the possibility of deployment. In real networks we need
> tcpdump, netflow, diagnostics, other debugging tools. Encapsulation
> works best when it does not require changes to the encapsulated
> packet. Besides that if your willing to change TCP for this use, why
> not just go a little farther and use SCTP/UDP which I believe already
> has a concept of shared congestion window amongst sub-flows?

You’re completely right. In fact I expect the performance of this TCP coupling to be quite similar to SCTP/UDP with multistreaming. The difference is in how this is implemented: it can, for example, be plugged underneath existing applications that use TCP, and it’s also a way to play with new TCP options that middleboxes wouldn’t like. It’s a small overhead method - TCP code has been optimized to death, SCTP comes with some overheads such as the CRC checksum (but it also has some more benefits).

> That will break our ability to use NIC offload with TCP/UDP
> (segmentation offload for instance). Everything, including many
> performance optimizations, is already designed around the requirement
> that the TCP checksum must be set. Trying to undo that requirement is
> actually a complication, not a simplification in practice.

Interesting… here I lack knowledge about offloading. Are you saying that GUE doesn’t have that problem? That would make it a more attractive encapsulation option then.

Cheers,
Michael

---

From: Tom Herbert <[email-masked]>
Date: Fri, 25 Mar 2016 10:58:51 -0700

> I have, as I wrote this draft. It struck me as something that isn’t really necessary just for the purpose of what TCP-in-UDP is trying to achieve. In particular, other than GUE (if I got it correct…
Then you are redefining TCP. You can do that, but this substantially

Tom

---

From: Tom Herbert <[email-masked]>
Date: Fri, 25 Mar 2016 11:51:55 -0700

> Interesting… here I lack knowledge about offloading. Are you saying that GUE doesn’t have that problem? That would make it a more attractive encapsulation option then.

There's been a lot of work in this area since encapsulation over UDP
is very hot as a solution for tunneling. So we are working on
solutions that are generic across all flavors of foo-over-udp for L2
and L3 encapsulation. I believe these solutions should be applicable
to L4 encapsulation also (e.g. TCP/UDP). What we don't want is HW
vendors endlessly adding protocol specific support in HW for all these
foo-over-udp. With some care, we should be able to design new
encapsulations and generic HW support that avoid the need for
protocol-specific awareness in HW.

Tom

---

From: Michael Welzl <[email-masked]>
Date: Fri, 25 Mar 2016 19:53:54 +0100

> On 25. mar. 2016, at 19.51, Tom Herbert <[email-masked]> wrote:
> 

Understood, makes perfect sense to me.

Cheers,
Michael