Subject: Re: [tsvwg] transport protocol feedback overhead

From: "Gengxuesong (Geng Xuesong)" <[email-masked]>
Date: Thu, 6 Apr 2017 01:41:45 +0000

Hi Bob,

I think it is an interesting idea to solve the problem, but I wonder that whether doing this will cause other problem, because the ACK carries more information than arrival time.
For example if an ACK with congestion information arrives later than before, then the congestion response will be delayed either, which may do harm to the congested network.

Best Regards,

Xuesong

-----Original Message-----
From: tsvwg [mailto:[email-masked]] On Behalf Of Bob Briscoe
Sent: Friday, March 31, 2017 4:18 AM
To: Ingemar Johansson S; tsvwg IETF list
Subject: [tsvwg] transport protocol feedback overhead

Ingemar, list,

I seemed to get blocked from talking after I was trying to be polite - letting you finish your talk...

You described one problem:
  (A) In mobile (and other networks)  upstream ACK load is a problem.

There is a complementary problem too:
(B) Capacity variation is getting huge as radio wavelength increases. So it's useful for the sender to watch the timing of the ACKs to be able to track changes in capacity as fast as possible. It's also useful to be able to vary the spacing between the packets you send to probe for increases in capacity by watching the ACK spacing in response.

These two are in tension - (A) wants less ACKs, (B) wants them all.

A possible solution, is a data structure in a stretch ACK to be able to report the time of arrival of all the previous data packets (ie. the packets that the stretch ACK is ACKing.

Even tho this info will arrive at the sender later than it would have done if the ACKs had not been suppressed, as long as you are still getting a reasonable number of ACKs every RTT, it resolves the tension between A and B.

Thoughts?
Does this idea already exist?

Bob

PS. Apologies for dropping an innovative idea into a standards meeting. 
This probably ought to be a research paper first.

--
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

---

From: Jana Iyengar <[email-masked]>
Date: Thu, 6 Apr 2017 17:45:40 -0700

Thanks for the presentation Ingemar; unfortunately I missed this at the
IETF. As one of the authors of RFC 5690, I'm familiar with the pitfalls of
doing ack CC, but I think we may be able to get around them in QUIC since
there's explicit ack information in QUIC. We are experimenting with this in
QUIC, and we'll be happy to bring data when we have some.

On Fri, Mar 31, 2017 at 10:35 AM, Ingemar Johansson S <
[email-masked]> wrote:

> Thanks for the details. It may be the case that it can be tricky to get a
> decent understanding of what is possible in terms of ACK rate reduction for
> TCP. The middlebox behavior to discard everything except the last ACK has
> also been proposed in the RAN2 WG in 3GPP.
>
>
>
> /Ingemar
>
>
>
> *From:* Yuchung Cheng [mailto:[email-masked]]
> *Sent:* den 31 mars 2017 00:17
> *To:* Michael Welzl <[email-masked]>
> *Cc:* Ingemar Johansson S <[email-masked]>; tsvwg IETF
> list <[email-masked]>; [email-masked] Extensions <[email-masked]>
> *Subject:* Re: [tsvwg] TCP ACK-CC (re: Transport protocol feedback
> overhead )
>
>
>
> regarding the question in the slide 5.
>
> "Is immediate ACK needed for out of order segments if RACK is used ?"
>
>
>
> The answer is both yes and no: a timely feedback is appreciated (yes). But
> RACK does not need 100 DUPACKs each telling one more packet is delivered
> out of order (no). In other words if the ACK is timely and precise about
> the delivery process, RACK would work well (so does BBR). However this can
> go terribly wrong if the middle-box sends only the last ACK/SACK. It has
> happened in one major cloud platform.
>
>
>
> On Thu, Mar 30, 2017 at 2:14 PM, Michael Welzl <[email-masked]> wrote:
>
> Hi,
>
>
>
> Thanks Ingemar for your interesting presentation today:
>
> https://www.ietf.org/proceedings/98/slides/slides-
> 98-tsvwg-sessb-7-transport-protocol-feedback-overhead-
> issues-and-solutions-00.pdf
>
>
>
> I wanted to share with you, and the list, that David Ros  (one of the
> original authors of ACK-CC, RFC 5690) and I once advised a master student
> to:
>
> - implement ACK-CC
>
> - test it
>
> - analyze all RFC 2119 - style keywords in RFC 5690, to see how they
> should be changed (if at all) in a possible update
>
>
>
> Back then, our thinking was that we could write a paper about it, and try
> to get this deployed as a way of doing the Experiment and moving this to PS
> status.
>
> However, we were less and less sure about the real value of this - whether
> actual networks need this? Then, we had less and less time available and at
> some point dropped the ball.
>
> I of course found it interesting to hear that it *is* a real problem, even
> today.
>
>
>
> Anyway: this master thesis is available here:  http://heim.ifi.uio.no/
> michawe/teaching/dipls/marius-olsen/mastersthesis-mariusno.pdf
>
> and I think I can safely say on behalf of David, Marius (the master
> student who did this) and myself that we’re happy if someone picks this up
> and continues the work.
>
> It would require some digging, but I might also still have the code lying
> around somewhere - for some outdated version of Linux…
>
>
>
> Cheers,
>
> Michael
>
>
>
>
>

---

From: Ingemar Johansson S <[email-masked]>
Date: Fri, 7 Apr 2017 06:43:21 +0000

Hi
Thanks, looking forward to see the results of your experiments.
Important to understand is that the goal is not “reduced ACK rate at any cost”. Certain arrangements will need a higher ACK rate than others for instance depending on congestion control specifics.

But hopefully the ACK rate will decrease in a statistical sense and that is good, furthermore work on this will (possibly in an ideal world) send a signal that the ACKs that traverse the network are needed for one reason or the other and that middleboxes should not try to tamper with  these ACKs , call me a dreamer…

/Ingemar

From: Jana Iyengar [mailto:[email-masked]]
Sent: den 7 april 2017 02:46
To: Ingemar Johansson S <[email-masked]>
Cc: Yuchung Cheng <[email-masked]>; Michael Welzl <[email-masked]>; [email-masked] Extensions <[email-masked]>; tsvwg IETF list <[email-masked]>; Ian Swett <[email-masked]>
Subject: Re: [tsvwg] TCP ACK-CC (re: Transport protocol feedback overhead )

Thanks for the presentation Ingemar; unfortunately I missed this at the IETF. As one of the authors of RFC 5690, I'm familiar with the pitfalls of doing ack CC, but I think we may be able to get around them in QUIC since there's explicit ack information in QUIC. We are experimenting with this in QUIC, and we'll be happy to bring data when we have some.

On Fri, Mar 31, 2017 at 10:35 AM, Ingemar Johansson S <[email-masked]<mailto:[email-masked]>> wrote:
Thanks for the details. It may be the case that it can be tricky to get a decent understanding of what is possible in terms of ACK rate reduction for TCP. The middlebox behavior to discard everything except the last ACK has also been proposed in the RAN2 WG in 3GPP.

/Ingemar

From: Yuchung Cheng [mailto:[email-masked]<mailto:[email-masked]>]
Sent: den 31 mars 2017 00:17
To: Michael Welzl <[email-masked]<mailto:[email-masked]>>
Cc: Ingemar Johansson S <[email-masked]<mailto:[email-masked]>>; tsvwg IETF list <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]> Extensions <[email-masked]<mailto:[email-masked]>>
Subject: Re: [tsvwg] TCP ACK-CC (re: Transport protocol feedback overhead )

The answer is both yes and no: a timely feedback is appreciated (yes). But RACK does not need 100 DUPACKs each telling one more packet is delivered out of order (no). In other words if the ACK is timely and precise about the delivery process, RACK would work well (so does BBR). However this can go terribly wrong if the middle-box sends only the last ACK/SACK. It has happened in one major cloud platform.

On Thu, Mar 30, 2017 at 2:14 PM, Michael Welzl <[email-masked]<mailto:[email-masked]>> wrote:
Hi,

Thanks Ingemar for your interesting presentation today:
https://www.ietf.org/proceedings/98/slides/slides-98-tsvwg-sessb-7-transport-protocol-feedback-overhead-issues-and-solutions-00.pdf

I wanted to share with you, and the list, that David Ros  (one of the original authors of ACK-CC, RFC 5690) and I once advised a master student to:
- implement ACK-CC
- test it
- analyze all RFC 2119 - style keywords in RFC 5690, to see how they should be changed (if at all) in a possible update

Back then, our thinking was that we could write a paper about it, and try to get this deployed as a way of doing the Experiment and moving this to PS status.
However, we were less and less sure about the real value of this - whether actual networks need this? Then, we had less and less time available and at some point dropped the ball.
I of course found it interesting to hear that it *is* a real problem, even today.

Anyway: this master thesis is available here:  http://heim.ifi.uio.no/michawe/teaching/dipls/marius-olsen/mastersthesis-mariusno.pdf
and I think I can safely say on behalf of David, Marius (the master student who did this) and myself that we’re happy if someone picks this up and continues the work.
It would require some digging, but I might also still have the code lying around somewhere - for some outdated version of Linux…

Cheers,
Michael

---

From: Ingemar Johansson S <[email-masked]>
Date: Fri, 7 Apr 2017 06:50:34 +0000

Hi

Sure, ACKs carry a lot of information, this is the strongest argument against middlebox interaction BTW. 
Also the intention with ACK rate reduction is not "at any cost". One need to strike a balance between ACK rate and performance where the tolerance against reduced performance should be very low.

/Ingemar

> From: Gengxuesong (Geng Xuesong) [mailto:[email-masked]]
> Sent: den 6 april 2017 03:42
> To: Bob Briscoe <[email-masked]>; Ingemar Johansson S
> <[email-masked]>; tsvwg IETF list <[email-masked]>
> Subject: RE: [tsvwg] transport protocol feedback overhead
> 
> Hi Bob,
> 
> 
> I think it is an interesting idea to solve the problem, but I wonder that
> whether doing this will cause other problem, because the ACK carries more
> information than arrival time.
> For example if an ACK with congestion information arrives later than before,
> then the congestion response will be delayed either, which may do harm to
> the congested network.
> 
> 
> Best Regards,
> 
> Xuesong
> 
> (B) Capacity variation is getting huge as radio wavelength increases. So it's
> useful for the sender to watch the timing of the ACKs to be able to track
> changes in capacity as fast as possible. It's also useful to be able to vary the
> spacing between the packets you send to probe for increases in capacity by
> watching the ACK spacing in response.
> 
> A possible solution, is a data structure in a stretch ACK to be able to report
> the time of arrival of all the previous data packets (ie. the packets that the
> stretch ACK is ACKing.
> 
> Even tho this info will arrive at the sender later than it would have done if the
> ACKs had not been suppressed, as long as you are still getting a reasonable
> number of ACKs every RTT, it resolves the tension between A and B.
> 
> Thoughts?

---

From: "Gengxuesong (Geng Xuesong)" <[email-masked]>
Date: Fri, 7 Apr 2017 07:12:05 +0000

Hi Ingemar,

Thank you for our response.
It is reasonable to find a point that can reduce the ACK rate without hurting its performance, but I also wonder that whether ACK carries too much information which can be considered to give to the middlebox.
For example the congestion notification generated in the middle box can be sent to the source directly without going through the receiver, just like QCN in Layer 2.
I don't know whether this also belongs to the "middlebox interaction" you have mentioned : )

Best Regards,

Xuesong

-----Original Message-----
From: Ingemar Johansson S [mailto:[email-masked]] 
Sent: Friday, April 07, 2017 2:51 PM
To: Gengxuesong (Geng Xuesong); Bob Briscoe; tsvwg IETF list
Subject: RE: [tsvwg] transport protocol feedback overhead

Hi

/Ingemar

> I think it is an interesting idea to solve the problem, but I wonder 
> that whether doing this will cause other problem, because the ACK 
> carries more information than arrival time.
> For example if an ACK with congestion information arrives later than 
> before, then the congestion response will be delayed either, which may 
> do harm to the congested network.
> 
> 
> Best Regards,
> 
> Xuesong
> 
> (B) Capacity variation is getting huge as radio wavelength increases. 
> So it's useful for the sender to watch the timing of the ACKs to be 
> able to track changes in capacity as fast as possible. It's also 
> useful to be able to vary the spacing between the packets you send to 
> probe for increases in capacity by watching the ACK spacing in response.
> 
> A possible solution, is a data structure in a stretch ACK to be able 
> to report the time of arrival of all the previous data packets (ie. 
> the packets that the stretch ACK is ACKing.
> 
> Even tho this info will arrive at the sender later than it would have 
> done if the ACKs had not been suppressed, as long as you are still 
> getting a reasonable number of ACKs every RTT, it resolves the tension between A and B.
> 
> Thoughts?

---

From: Michael Welzl <[email-masked]>
Date: Fri, 7 Apr 2017 09:44:07 +0200

All,

I'm cc'ing ICCRG because these are truly research musings IMO. Maybe, if folks agree, we can truly move this thread and drop the tsvwg list from recipients?

A thought:
One reason we didn't continue our own work on this (the master thesis I mentioned previously) was that it wasn't 100% clear to us whether this is a *real* problem - is it a problem worth solving? How often does it occur? How problematic is it?

I think Ingemar made a good point in his presentation about the problem being real. Still, often ACKs are mixed with payload traffic. Consider the following scenario:
Say, I have a thin uplink.
In parallel, I download a file, and I upload a file (just as a simple model).
Both take long, there is congestion, packets are dropped.

=> should the rate of outgoing payload be reduced, or should the ACK rate be reduced? Or both?

Given that ACKs tend to be smaller, losing too many ACKs could be quite harmful, and the Internet has survived very long without ACK rate reduction, I'd tend to think the payload should be the first to react.
So then, a scheme that changes the ACK rate could try to figure out if it truly was the ACKs that created congestion.

This could be done by checking the impact of one's actions with some heuristics. For example: if I increase the rate, does delay increase with it, and does it decrease again if I decrease the rate?
What if, different from a "normal" congestion control reaction where senders try to back off immediately upon any form of congestion indication, ACK rate reduction would do such tests for 2-3 round trips before actually backing off?  This would avoid a situation where, in my example, both the download and the upload traffic would back off together (which might be overdoing it).

Just some thoughts - as I said, research ideas.

Cheers,
Michael

> On 07 Apr 2017, at 09:12, Gengxuesong (Geng Xuesong) <[email-masked]> wrote:
> 
> Hi Ingemar,
>

---

From: Ingemar Johansson S <[email-masked]>
Date: Fri, 7 Apr 2017 07:50:41 +0000

Hi

As I see it, these kinds of middlebox generated feedback ideas is probably a different topic and I believe that can be good to keep the two topics (middlebox generated ACKs, and ACK rate in end hosts) separated, because the middlebox ACK involves other concerns around authentication et.c.
My main interest for now is in getting the ACK rate reduced where possible and also some kind of documentation that describes what is done already today.
 
/Ingemar

> Sent: den 7 april 2017 09:12
> To: Ingemar Johansson S <[email-masked]>; Bob Briscoe
> <[email-masked]>; tsvwg IETF list <[email-masked]>
> It is reasonable to find a point that can reduce the ACK rate without hurting
> its performance, but I also wonder that whether ACK carries too much
> information which can be considered to give to the middlebox.
> For example the congestion notification generated in the middle box can be
> sent to the source directly without going through the receiver, just like QCN
> in Layer 2.
> Also the intention with ACK rate reduction is not "at any cost". One need to
> strike a balance between ACK rate and performance where the tolerance
> against reduced performance should be very low.
> 
> /Ingemar
>

---

From: "Gengxuesong (Geng Xuesong)" <[email-masked]>
Date: Fri, 7 Apr 2017 08:12:07 +0000

Hi Ingemar,

Actually I think they can be related.
Because there are some functions in ACK which require rapid response. If we can remove some of these functions out of ACK, we can reduce the ACK rate in the host more easily.
But yes, we can also discuss these two topics separately, see what can be done in the ACK rate with current ACK functions.

Best Regards,
Xuesong

-----Original Message-----
From: Ingemar Johansson S [mailto:[email-masked]] 
Sent: Friday, April 07, 2017 3:51 PM
To: Gengxuesong (Geng Xuesong); Bob Briscoe; tsvwg IETF list
Cc: Ingemar Johansson S
Subject: RE: [tsvwg] transport protocol feedback overhead

Hi

/Ingemar

> To: Ingemar Johansson S <[email-masked]>; Bob 
> Briscoe <[email-masked]>; tsvwg IETF list <[email-masked]>
> It is reasonable to find a point that can reduce the ACK rate without 
> hurting its performance, but I also wonder that whether ACK carries 
> too much information which can be considered to give to the middlebox.
> For example the congestion notification generated in the middle box 
> can be sent to the source directly without going through the receiver, 
> just like QCN in Layer 2.
> Also the intention with ACK rate reduction is not "at any cost". One 
> need to strike a balance between ACK rate and performance where the 
> tolerance against reduced performance should be very low.
> 
> /Ingemar
> 
> > before, then the congestion response will be delayed either, which 
> > may do harm to the congested network.
> >
> >
> > Best Regards,
> >
> > Xuesong
> >
> > useful to be able to vary the spacing between the packets you send 
> > to probe for increases in capacity by watching the ACK spacing in response.
> >
> > Even tho this info will arrive at the sender later than it would 
> > have done if the ACKs had not been suppressed, as long as you are 
> > still getting a reasonable number of ACKs every RTT, it resolves the 
> > tension
> between A and B.
> >
> > Thoughts?

---

From: Bob Briscoe <[email-masked]>
Date: Fri, 7 Apr 2017 16:39:22 +0100

Michael,

1/ Real problem?

I agree that we should first establish whether ACK load is a problem.
I was responding to Ingemar assuming he believed this was true, given 
that was his presentation.

I can understand that ACKs are not so much a problem because of the bits 
they carry. More because they consume transmission opportunities on 
wireless links. Also, because they consume processor cycles and 
interrupts on hosts (which is why ACK thinning is done in DCs).

2/ tsvwg/iccrg?

Actually, as Ingemar mentioned in his slides, this is potentially 
something to consider for AccECN, which is a WG doc in tcpm now.

It would at least be useful to consider how this timing info might be 
added later to the same AccECN TCP option. I mean, perhaps introduce a 
placeholder so that a further draft can be written later to add timing 
info to the same TCP option.

Note: AccECN is solely about ECN, whereas arrival timing info can be 
relevant irrespective of ECN support, so they might not fit well. 
However, in one respect the two functions fit together nicely because 
like the AccECN option, timing info is truly optional - feedback can 
still work (minimally) if the TCP option does not traverse a path. 
Amortizing both into one option should help deployment of both.

3/ Feedback delay

To answer Xuesong's initial point, I only proposed including timing info 
in ACKs /if/ the ACK rate needed to be reduced. Of course missing out 
some ACKs introduces a delay before another ACK arrives, and delays any 
timing info it contains. But if ACKs are going to be thinned/stretched, 
adding timing info about intervening arrivals is better than not adding 
it. If there are, say, W=100 packets in a window,:
* ACKing 1 in 2 packets causes the worst total feedback delay to be 101% 
of an RTT.
* ACKing 1 in 8 packets causes the worst total feedback delay to be 107% 
of an RTT.
More generally,
* ACKing 1 in d packets causes the worst total feedback delay to be 
1+(d-1)/W of an RTT.
Given ACK rate is generally only a problem when W is large, the extra 
delay could be of concern, but it should not be significant.

4/ Middle-to-sender feedback

Re. Xuesong's second point,... been there, doesn't work. See 
https://www.ietf.org/mail-archive/web/tsvwg/current/msg10843.html

This mode of feedback appeared on the tsvwg list in 2006, 2011, 2017... 
On this basis it should come up again in about 2023 :)

Bob

On 07/04/17 08:44, Michael Welzl wrote:
> _______________________________________________
> iccrg mailing list
> [email-masked]
> https://www.irtf.org/mailman/listinfo/iccrg