Subject: Re: [iccrg] [Stackevo-discuss] New Version Notification for draft-welzl-irtf-iccrg-tcp-in-udp-00.txt

From: Michael Welzl <[email-masked]>
Date: Fri, 25 Mar 2016 09:53:57 +0100

So now I’m answering the congestion control bits here. I’ll reorder things a little to more effectively answer points that belong together, I hope you don’t mind.

Side note: while this is about TCP using multiple paths, so far we have left MPTCP out of the discussion, and that probably makes sense. It does pretty clearly reflect "one congestion controller per path” thinking, I would say, and that makes it come from a different angle than what we’re discussing here: assuming a network of multiple paths, not a single path, for a single end-to-end congestion controller.

> On 24. mar. 2016, at 18.46, Joe Touch <[email-masked]> wrote:
> 
> 
> 
> On 3/24/2016 1:20 AM, Michael Welzl wrote:
>> 
>>> On 24. mar. 2016, at 00.52, Joe Touch <[email-masked]> wrote:
> ...
> 
>>>>> I can't imagine what combined congestion control of TCP would mean
>>>>> inside the network - we had a hard enough time defining it for either
>>>>> TCB sharing or the Congestion Manager just at the endpoints.
>>>> 
>>>> I don’t get this either; it sounds like a misunderstanding?
>>>> 
>>>> So the encapsulation here is just meant to force packets to take the 
>>>> same path - else the Congestion Manager may be doing something
>>>> pretty wrong.
>>> 
>>> The CM is supposed to react to E2E congestion, which is the sum of all
>>> effects of the different paths that packets can take.
>> 
>> I’ll have to disagree. The CM is one congestion control instance, 
>> and E2E TCP-like congestion control is based on the assumption of a
>> single path. Things go wrong if this assumption is broken, e.g. when
>> packets from a single connection are sent in different directions
>> inside the network. This is not only due to possible reordering, but
>> also because e.g. a congestion event on path 1 would make the
>> controller reduce its rate, even if path 2 still has 3 times more
>> capacity available.
> 
> TCP has always handled reordering.

In one way or another, yes…  I (and I’m sure, you) know of various attempts to deal with it; first off, stretching the threshold to allow more than 3 DupACKs, which is a trade-off between allowing more reordering and making the response slower. Second, trying to survive even when you got it wrong - i.e., seeing spurious loss events, and then having to undo the response; yes, good TCP implementations can cope with these things to some degree, but repeatedly seeing and undoing spurious loss events isn’t exactly what I’d call TCP’s normal operation condition. It’s also implementation dependent - this Ph.D. thesis:
http://folk.uio.no/paalh/students/DominikKaspar-phd.pdf
finds that Linux TCP can handle reordering better than a fully standard-conforming implementation would.

Bottom line, I agree TCP handles it, but I wouldn’t say these are its “normal” (preferred) operation conditions. Anyway, I mainly mentioned it because it’s the more prominent problem that one gets when using multiple paths. The other one that I mentioned above seems more severe to me. See below:

> TCP congestion control treats the network as a system - if the net as a
> whole drops packets, it backs off. If the net as a whole does not, it
> increases. That's consistent with TCP CC. You don't need to assume a
> single path.
(..)
> TCP CC absolutely allows us to assume different paths - it reacts to the
> *network* as a transfer mechanism, i.e., where the "path" is through the
> blob we call a network. The only real problem is when that blob's
> properties vary, but that's always a problem for every feedback system.

so then how do you solve the 2-path problem I explained above? Let me make this clearer, with a simple, very static example problem (the only dynamics here come from a single TCP connection):
Consider an empty network. Path 1 has capacity X. Path 2 has capacity 3*X. Some device in the network decides to round-robin schedule packets from a single TCP connection across paths 1 and 2.
Congestion will first appear on path 1. TCP will react, halving its cwnd. Most of the capacity of path 2 will always remain unused that way.

How do you solve this with e2e congestion control?

>> (*) Researchy side note here: I’m all for fixing this, but this
>> would require putting congestion controls inside the network,
> 
> No, it really doesn't. It just means we assume that the net is the
> "path", not that we know or have control over a specific set of links.

The only solution I can see to my problem above is not to schedule traffic as I described it. Obviously the scheduler should send more packets on path 2 than on path 1. Doing this correctly requires knowing the capacities of these links. Considering a more dynamic network situation, it requires knowledge about where the bottleneck currently is, and how much capacity is available per link. This is a control loop. If you put such a control loop in the network, you may want to give it a different name than me, but this is what I meant when I said “this would require putting congestion controls inside the network”.

> NOTE: threads along these lines have been popping up on V6OPS recently
> too; it'd be useful to check there. I'm not the only one who thinks we
> should stop trying to act like we can pin paths merely by picking IP
> addresses, flow labels, or doing DPI - all we ought to assume is that
> everything within a flow is "treated the same way", which does not mean
> that every packet is treated IDENTICALLY.

As per my example above, this would mean that you’d need something quite similar to congestion control inside the network, or the available capacity for packets on all these paths must be equal, all the way to the receiver - else you’ll end up underutilizing your network.

Cheers,
Michael

---

From: John Leslie <[email-masked]>
Date: Fri, 25 Mar 2016 06:54:48 -0400

Michael Welzl <[email-masked]> wrote:
>...
> so then how do you solve the 2-path problem I explained above? ...
> Consider an empty network. Path 1 has capacity X. Path 2 has capacity
> 3*X. Some device in the network decides to round-robin schedule packets
> from a single TCP connection across paths 1 and 2.
> Congestion will first appear on path 1. TCP will react, halving its
> cwnd. Most of the capacity of path 2 will always remain unused that way.
> 

   Obviously, as you design the example, you don't.

   Neither the e2e sender nor the e2e receiver has enough knowledge if
some device along the path decides to switch paths without notice.

   (Fortunately, we don't have to discuss the case where the receiver
get notice of which path was used but the sender must guess which
path will be used!)

>... 
> The only solution I can see to my problem above is not to schedule
> traffic as I described it. Obviously the scheduler should send more
> packets on path 2 than on path 1.

   Whadaya mean, "the" scheduler? I see at least two of them here!

> Doing this correctly requires knowing the capacities of these links.

   Indeed. If the round-robin scheduler chooses to ignore the difference
in capacities, the TCP scheduler can't expect to overcome that.

   (There are operators which do this sort of thing: usually they manually
adjust things so the capacities are "close enough" and make _some_ effort
to avoid this sub-network becoming a bottleneck.)

> Considering a more dynamic network situation, it requires knowledge
> about where the bottleneck currently is, and how much capacity is
> available per link. This is a control loop.

   We have been spoiled by bottlenecks being mostly stable. They won't
always be.

   The question of where a bottleneck "currently is" isn't really knowable.
They tend (in 2015) to be stable enough that it _seems_ knowable; but
even "currently" doesn't really mean what we think it does.

   Operators find that certain "bottlenecks" can be quickly fixed by
shifting/adding capacity, while others resist that solution. Consequently,
they play _that_ whack-a-mole game; and this tends to drive "current"
bottlenecks close to the end-users. 

> If you put such a control loop in the network, you may want to give it
> a different name than me, but this is what I meant when I said
> ???this would require putting congestion controls inside the network???.

   I suspect that "a different name" would be very helpful...

   There is a lot of history of "congestion control" in networks, mostly
based upon admission-control. This seems to me less than compatible with
the TCP model of congestion control.

> As per my example above, this would mean that you???d need something
> quite similar to congestion control inside the network, or the available
> capacity for packets on all these paths must be equal, all the way to
> the receiver - else you???ll end up underutilizing your network.

   Operators _don't_ have the same horror about "underutilizing" that
you do.

   It turns out to be very practical to drive the bottlenecks close to
the endpoints; and aggregation tends to ameliorate under-utilization...

--
John Leslie <[email-masked]>

---

From: Michael Welzl <[email-masked]>
Date: Fri, 25 Mar 2016 12:48:23 +0100

> On 25. mar. 2016, at 11.54, John Leslie <[email-masked]> wrote:
> 

Sorry, I meant the in-network scheduler that decides which of the two paths packets will take.

> to avoid this sub-network becoming a bottleneck.)

Interesting!

> the TCP model of congestion control.

…though these were just research musings anyway. I’m not at all proposing to design that in-network control loop with this draft.

> the endpoints; and aggregation tends to ameliorate under-utilization…

Sure, that all makes sense to me - but you’re talking about operators dealing with today’s system. I don’t think that we want to design a congestion control based on the assumption that underutilizing won’t matter anyway?

Cheers,
Michael

---

From: John Leslie <[email-masked]>
Date: Fri, 25 Mar 2016 08:35:22 -0400

Michael Welzl <[email-masked]> wrote:
>> the endpoints; and aggregation tends to ameliorate under-utilization
> 
> Sure, that all makes sense to me - but you???re talking about operators
> dealing with today???s system. I don???t think that we want to design
> a congestion control based on the assumption that underutilizing
> won???t matter anyway?

   That's not what I meant to say...

   I was trying to talk about what we're really experiencing today. I'm
not at all sure there's any end-to-end improvement available from
optimizing the balance of multi-path forwarding not under the control
of one of the endpoints.

   IMHO, there _are_ issues where "TCP Congestion Control" is a bit
brain-dead; and could be improved by tracking the capacity of the path(s)
available to it.

   But its failure to utilize path capacity stems from ramp-up problems
and its "steady-state" insistance on sawtoothing.

   I'd like us to work on getting better congestion signals and making
better use of them.

---

From: Michael Welzl <[email-masked]>
Date: Fri, 25 Mar 2016 14:15:30 +0100

> On 25. mar. 2016, at 13.35, John Leslie <[email-masked]> wrote:
> 

Well - we’re in line here: one easy way to get a better congestion signal is by not ignoring that there already is an ongoing transfer to the same destination. Ramp up is exactly one of the things that gets better than: you know immediately from the other flow how fast you can go. That’s one of the things that this congestion control coupling can give you.

Cheers,
Michael

---

From: Michael Welzl <[email-masked]>
Date: Fri, 25 Mar 2016 17:57:01 +0100

Please discuss this draft on the ICCRG list. I’ll keep stackevo-discuss in cc just this once as a reminder to folks here that this has moved, but please remove it in your response. Let’s try to cleanly transition to ICCRG if we can.

About your email, my general answer is: you seem to be reading more into this than it is.

You mention reliability, applicability to video streaming, etc.: this is still “normal” TCP. In fact a design goal was to minimize the changes that we need to do to it, so it’s a relatively easy code change. The intention is only to be able to combine the congestion controllers of multiple flows, as has been suggested before (CM, Ensemble sharing, ..). The outcome is that multiple flows are closer to being like one flow, and there should be some benefits there in terms of new flows immediately being able to use a larger cwnd, priority-based cwnd share assignment etc.

Since you mention tunnels, issues with traceroute, overlays, etc.: this is an e2e encapsulation. Same TCP source and destination endpoints as before.

Does it require rendez-vous points: it requires a well-known UDP port. The idea is that a client checks using UDP if the server is TCP-in-UDP capable (listening on the UDP port), and if not, falls back to TCP. This is achieved by “happy eyeballing” (sending out packets of both types in one go).

Cheers,
Michael

> On 25. mar. 2016, at 16.04, Alexandre Petrescu <[email-masked]> wrote:
> 
> 
> 
> Le 25/03/2016 09:12, Michael Welzl a écrit :
>> 
>>>>> Having one uniform layer at which multiplexing occurs makes
>>>>> things easy. Having two means you have to wonder what the
>>>>> other is doing all the time and whether the two correlate.
>>>> 
>>>> Is this a real practical problem related to this draft?
>>> 
>>> It is a real practical problem anytime you tunnel.
>> 
>> Even when the method avoids that packets grow in size?
> 
> The size of the tunnelled packets may not be as big as expected problem.
> For example, additional 40bytes on wireless are not that bad as
> initially expected, simply because the total size is already around
> 1280bytes before tunneling.
> 
> On another hand, this problem of multiplexing at multiple levels makes
> it that there is no 'traceroute' program that meaningfully reports
> number of hops through tunnels.  This is a universal tool for network debugging, maybe as loved as ping is.
> 
> Overlay networks grow for a while and then get levelled down into
> 'native' multiplexing level if I can say so.  For example IPv6 tunnels
> over an IPv4 world moved to IPv6 native (3ffe disappeared a while back
> and 6to4 recently; each had a different way of relating two multiplexing
> levels).
> 
> For TCP in UDP there would be a myriad ways of mixing the slow
> intelligence of TCP with the quick dumbness of UDP.  After trying them
> all would one come back to TCP and UDP side-by-side as before?  Why not?
> 
> Is TCP-in-UDP as reliable as TCP?  Or is it more subject to packet loss
> and longer retries?
> 
> Is TCP-in-UDP useful for video streaming from constrained devices, as
> useful as UDP is?
> 
> Does TCP-in-UDP require rendez-vous points?
> 
> Alex
> 
>> I’m surprised but I’m not going to debate this further: you provided
>> a useful pointer (the tunnels draft in INTAREA), I’ll have to take a
>> look and try to understand the problem. Anyway I agree that a
>> solution like the v6 flow labe that Tom Herbert suggested is nicer; I
>> have nothing at all against that and now plan to incorporate it in
>> the next version of the draft, for the v6 case.
>> 
>> As for the congestion control related bits of your email, I made a
>> mistake: I started this off saying “let’s discuss this in ICCRG”,
>> but then I follow up a discussion here on stackevo. This is purely
>> about congestion control, so it really belongs there. My next email
>> in this matter will go to you, cc iccrg.
>> 
>> Cheers, Michael
>> 
>> _______________________________________________ Stackevo-discuss
>> mailing list [email-masked]
>> https://www.iab.org/mailman/listinfo/stackevo-discuss
>> 
> 
> _______________________________________________
> Stackevo-discuss mailing list
> [email-masked]
> https://www.iab.org/mailman/listinfo/stackevo-discuss

---

From: Alexandre Petrescu <[email-masked]>
Date: Fri, 25 Mar 2016 18:16:08 +0100

Le 25/03/2016 17:57, Michael Welzl a écrit :
> Please discuss this draft on the ICCRG list. I’ll keep
> stackevo-discuss in cc just this once as a reminder to folks here
> that this has moved, but please remove it in your response. Let’s try
> to cleanly transition to ICCRG if we can.
>
>
> You mention reliability, applicability to video streaming, etc.: this
> is still “normal” TCP.

Ok for TCP, but the question was of TCP-in-UDP applicability to
video-streaming as opposed to UDP applicability in video-streaming.

> In fact a design goal was to minimize the changes that we need to do
> to it, so it’s a relatively easy code change.

I can agree.

> The intention is only to be able to combine the congestion
> controllers of multiple flows, as has been suggested before (CM,
> Ensemble sharing, ..). The outcome is that multiple flows are closer
>  to being like one flow, and there should be some benefits there in
> terms of new flows immediately being able to use a larger cwnd,
> priority-based cwnd share assignment etc.

I guess you mean multiple TCP flows.

> Since you mention tunnels, issues with traceroute, overlays, etc.:
> this is an e2e encapsulation. Same TCP source and destination
> endpoints as before.

But different couples of port numbers, right?  It would be
transport-layer encapsulation similar to network-layer encapsulation
which uses different couples of IP addresses.

> Does it require rendez-vous points: it requires a well-known UDP
> port.

Ok, so the TCP and UDP endpoitns would be on a same machine.  I was
thinking they were on different machines, and the machine owning the
well-known UDP port (rdv point) would multiplex to other TCP-only
machines which dont implement TCP-in-UDP.

> The idea is that a client checks using UDP if the server is
> TCP-in-UDP capable (listening on the UDP port), and if not, falls
> back to TCP.

There are a few questions here.

If a destination is not TCP-in-UDP capable will it tell it explicitely
to the source?  This is hard to do.

Or will the source decide based on the reception of a "UDP Port
Unreachable"?  If so, then you hit this need of the UDP layer to
communicate to the TCP layer: if  UDP port number is unavailable then
use this other TCP port number.  And there are so many ways to do that,
that at some point one may think it's better to leave them alone
separate, not one in the other.

> This is achieved by “happy eyeballing” (sending out packets of both
> types in one go).

I didnt know.  I will look.

Alex

>>>>>> Having one uniform layer at which multiplexing occurs
>>>>>> makes things easy. Having two means you have to wonder what
>>>>>> the other is doing all the time and whether the two
>>>>>> correlate.
>>>>>
>> The size of the tunnelled packets may not be as big as expected
>> problem. For example, additional 40bytes on wireless are not that
>> bad as initially expected, simply because the total size is already
>> around 1280bytes before tunneling.
>>
>> On another hand, this problem of multiplexing at multiple levels
>> makes it that there is no 'traceroute' program that meaningfully
>> reports number of hops through tunnels.  This is a universal tool
>> for network debugging, maybe as loved as ping is.
>>
>> 'native' multiplexing level if I can say so.  For example IPv6
>> tunnels over an IPv4 world moved to IPv6 native (3ffe disappeared a
>> while back and 6to4 recently; each had a different way of relating
>> two multiplexing levels).
>>
>> intelligence of TCP with the quick dumbness of UDP.  After trying
>> them all would one come back to TCP and UDP side-by-side as before?
>> Why not?
>>
>> Is TCP-in-UDP as reliable as TCP?  Or is it more subject to packet
>>  loss and longer retries?
>>
>> Is TCP-in-UDP useful for video streaming from constrained devices,
>>  as useful as UDP is?
>>
>>> I’m surprised but I’m not going to debate this further: you
>>> provided a useful pointer (the tunnels draft in INTAREA), I’ll
>>> have to take a look and try to understand the problem. Anyway I
>>> agree that a solution like the v6 flow labe that Tom Herbert
>>> suggested is nicer; I have nothing at all against that and now
>>> plan to incorporate it in the next version of the draft, for the
>>>  v6 case.
>>>
>>> As for the congestion control related bits of your email, I made
>>>  a mistake: I started this off saying “let’s discuss this in
>>> ICCRG”, but then I follow up a discussion here on stackevo. This
>>>  is purely about congestion control, so it really belongs there.
>>>  My next email in this matter will go to you, cc iccrg.
>>>
>>> Cheers, Michael
>>>

---

From: Michael Welzl <[email-masked]>
Date: Fri, 25 Mar 2016 18:43:51 +0100

> On 25. mar. 2016, at 18.16, Alexandre Petrescu <[email-masked]> wrote:
> 
> 
> 

It’s as applicable or non-applicable as TCP.

> I guess you mean multiple TCP flows.

Yes

> which uses different couples of IP addresses.

Exactly: you have one UDP port number pair, using a well-known port, that encapsulates several separate TCP port number pairs.
To the TCP’s on both sides, its normal port numbers are still there - they’re preserved, by first exchanging them on the SYN / SYN-ACK and then just “compressing” them into a connection ID.

> machines which dont implement TCP-in-UDP.

Aha! Well yes indeed, on the same machine. We change the TCP code on the client, we change the TCP code on the server, we involve no extra system.

> I didnt know.  I will look.

It’s described in section 4 of the draft:
https://tools.ietf.org/html/draft-welzl-irtf-iccrg-tcp-in-udp-00#section-4
The explanation begins under the bullet list, at the paragraph starting with "Unless it is known that…”.   Note to self: make a subsection for this in the next version…
We don’t wait for “UDP Port Unreachable”: we try both and use what comes back first.

Cheers,
Michael

---

From: Dirceu Cavendish <[email-masked]>
Date: Sat, 26 Mar 2016 14:58:37 +0000 (UTC)

I agree that TCP failure to utilize path capacity is related to its steady state sawtoothingbehavior (albeit Cubic improves on that). I've been experimenting with a path capacity 
estimated TCP for years, and I dont see a reason not to improve on sawtoothing.
Dirceu

    On Friday, March 25, 2016 10:44 AM, Michael Welzl <[email-masked]> wrote:

Yes

Cheers,
Michael

_______________________________________________
iccrg mailing list
[email-masked]
https://www.irtf.org/mailman/listinfo/iccrg