Subject: Re: [tcpm] [e2e] TCP HyStart patch deployment

From: Neal Cardwell <[email-masked]>
Date: Tue, 1 Sep 2015 20:25:23 -0400

On Tue, Sep 1, 2015 at 5:24 PM, Veaceslav ROMAN
<[email-masked]> wrote:
> DEALY algorithm supposed to compare average RTT of first 8 packets
> (curr_rtt) in the round with minRTT of the previous round

The Hystart delay-based algorithm compares the min RTT over the first
8 packets in a round trip (curr_rtt) with the minimum RTT (delay_min)
seen since the connection started, or the latest RTO.

> If I may help with traces from life LTE (not loaded) network, I'll do it with pleasure.

If you have a representative LTE packet trace (pcap) showing the
Hystart delay algorithm performing poorly, or a set of experiments
showing there are better parameters for the Hystart delay algorithm,
that would be interesting.

As Eric notes, the Hystart ACK Train algorithm is less interesting, as
Eric already discovered that one does not work well with pacing.

neal

---

From: Veaceslav ROMAN <[email-masked]>
Date: Wed, 2 Sep 2015 07:45:40 +0000

Thank you for prompt answer.
I've checked recent traces. 
The URI points to ...youtube.com, but the IP address points to 195.95.178.x which belongs to a hosting company.
Is it possible to not be patched ?

Veaceslav Roman
Technical and IT director
Orange Moldova S.A.
Fix: +37322575400
Mob: +37369198400
Fax: +37322575306

-----Original Message-----
From: Eric Dumazet [mailto:[email-masked]] 
Sent: Wednesday, 02 September 2015 00:49
To: Veaceslav ROMAN
Cc: Ingemar Johansson S; Piers O'Hanlon; Neal Cardwell; [email-masked]; Sangtae Ha; [email-masked]
Subject: Re: [e2e] TCP HyStart patch deployment

I really doubt Youtube exits slow start on ack train detection, since we disabled it on all Google fleet more than 16 months ago.

-static int hystart_detect __read_mostly = HYSTART_ACK_TRAIN | HYSTART_DELAY;
+/* Google-internal: we disable HYSTART_ACK_TRAIN because it exits slow
+ * start spuriously when used with the fq/pacing qdisc:
+ */
+static int hystart_detect __read_mostly = HYSTART_DELAY;

On Tue, Sep 1, 2015 at 2:24 PM, Veaceslav ROMAN <[email-masked]> wrote:
> I am impatient to see the results of your investigations.
> In real LTE network, seen systematic bad results. E.g. on downloading Youtube of 8 MB systematically exit slow start when Bytes-In-Flight ~50 KB, then slowly grow up to 500 KB until the end of the download and no drops at all (estimation using timestamps).
> On a test server, if Hystart switch off the average speed double (in LTE) , even though, of cause, drops appear. But Internet is not my test server.
>
> Such a pain that even begin to look at code. In particular, in original idea, which is good, the DEALY algorithm supposed to compare average RTT of first 8 packets (curr_rtt) in the round with minRTT of the previous round. But in current implementation is compared with min_delay which is continuously updated after the curr_rtt in the current round is computed (8 packets). Is it something here which I miss?
>
> Veaceslav Roman
> -----Original Message-----
> From: tcpm [mailto:[email-masked]] On Behalf Of Ingemar 
> Johansson S
> Sent: Monday, 22 June 2015 12:23
> To: Piers O'Hanlon; Neal Cardwell
> Cc: [email-masked]; Sangtae Ha; Ingemar Johansson S; Eric Dumazet; 
> [email-masked]
> Subject: Re: [tcpm] [e2e] TCP HyStart patch deployment
>
> Hi
>
> Sorry for the delay and thanks for the response. The 4.0.4 is the one that I have implemented. I have not involved any Linux networking list for the moment. That is perhaps a good idea once I get to understand this better.
>
> Currently I don't have any conclusive results that shows that there is a real issue with the current HyStart. Yes I can see a few cases where mainly the DELAY algorithm exits a bit early, and while one can argue that it is a bit too early, there seems anyway to be enough data in the RLC buffers to utilize the link well in most of the cases. It is only  in a few corner cases that I have seen issues with a very low link utilization.
>
> As I cannot right now find a larger set of cases where HyStart is a general issue it has not been possible to try out Neal's suggested modifications.
>
> I still have some more investigations to do after the vacation, this may show if there are general issues with HyStart.
>
> /Ingemar
>
>
>> -----Original Message-----
>> From: Piers O'Hanlon [mailto:[email-masked]]
>> Sent: den 5 juni 2015 19:02
>> To: Ingemar Johansson S
>> Cc: [email-masked]; [email-masked]
>> Subject: Re: [e2e] TCP HyStart patch deployment
>>
>> Hi Ingemar,
>>
>> Looking at linux-4.04 git code:
>> https://git.kernel.org/cgit/linux/kernel/git/stable/linux-
>> stable.git/tree/net/ipv4/tcp_cubic.c?id=refs/tags/v4.0.4#n403
>>
>> It seems the HYSTART_DELAY code has been tweaked into a somewhat 
>> similar direction (right shift 3 instead of the original 4) to the patch:
>>                           HYSTART_DELAY_THRESH(ca-
>> >delay_min >> 3)) {
>>
>> Although the ACK train detection does appear unchanged.
>>
>> It would seem that Hystart could have issues at very high rates as it 
>> does rely on a few default parameters that might not be suitable for 
>> such environments. I can also imagine that it may not work so well on 
>> WiFi with block ACKing as Hystart uses a fixed window of 8 packets to 
>> estimate current delay which may end up being one block of packets...
>>
>> Did you try asking this question on any Linux networking lists?
>>
>> Cheers,
>>
>> Piers.
>>
>> On 26 May 2015, at 12:38, Ingemar Johansson S wrote:
>>
>> > Hi
>> >
>> > Does anybody know if the TCP Cubic HyStart patch described in the 
>> > link
>> below is being used widely ?
>> > It does not seem to be implemented in the latest Linux code.
>> > http://patchwork.ozlabs.org/patch/85945/
>> >
>> > Regards
>> > Ingemar
>> > =================================
>> > Ingemar Johansson  M.Sc.
>> > Senior Researcher
>> >
>> > Ericsson AB
>> > Wireless Access Networks
>> > Labratoriegränd 11
>> > 971 28, Luleå, Sweden
>> > Phone +46-1071 43042
>> > SMS/MMS +46-73 078 3289
>> >
>> [email-masked]<mailto:ingemar.s.johansson@ericsson.
>> c
>> om>
>> > www.ericsson.com
>> >
>> > "No man has a good enough memory
>> >  to be a successful liar"
>> >    Abraham
>> Lincoln<http://www.brainyquote.com/quotes/authors/a/abraham_lincoln.h
>> tml>
>> > =================================
>> >
>> > _______________________________________________
>> > end2end-interest mailing list
>> > [email-masked]
>> > http://mailman.postel.org/mailman/listinfo/end2end-interest
>> > Contact [email-masked] for assistance.
>
> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm

---

From: Veaceslav ROMAN <[email-masked]>
Date: Wed, 2 Sep 2015 08:18:21 +0000

Thank you for prompt answer.
For youtube, of cause I do see only client side traces, and I can send immediately.
For the server side traces, can you suggest the minimum Linux kernel for which it will make sense to analyze traces?

Veaceslav Roman

-----Original Message-----
From: Neal Cardwell [mailto:[email-masked]] 
Sent: Wednesday, 02 September 2015 03:25
To: Veaceslav ROMAN
Cc: Ingemar Johansson S; Piers O'Hanlon; [email-masked]; Sangtae Ha; Eric Dumazet; [email-masked]
Subject: Re: [e2e] TCP HyStart patch deployment

The Hystart delay-based algorithm compares the min RTT over the first
8 packets in a round trip (curr_rtt) with the minimum RTT (delay_min) seen since the connection started, or the latest RTO.

> If I may help with traces from life LTE (not loaded) network, I'll do it with pleasure.

If you have a representative LTE packet trace (pcap) showing the Hystart delay algorithm performing poorly, or a set of experiments showing there are better parameters for the Hystart delay algorithm, that would be interesting.

As Eric notes, the Hystart ACK Train algorithm is less interesting, as Eric already discovered that one does not work well with pacing.

neal

---

From: Neal Cardwell <[email-masked]>
Date: Wed, 2 Sep 2015 08:25:44 -0400

> For youtube, of cause I do see only client side traces, and I can send immediately.

Great!

> For the server side traces, can you suggest the minimum Linux kernel for which it will make sense to analyze traces?

To be useful, the traces would need to be from a kernel that is
version 4.0 or newer, as the latest CUBIC changes were in v4.0. You
would also need to disable the ACK train detection with:

 echo 2 > /sys/module/tcp_cubic/parameters/hystart_detect

And then ideally you would run with the fq qdisc, to get pacing:
  https://lwn.net/Articles/565421/

neal

---

From: Neal Cardwell <[email-masked]>
Date: Wed, 2 Sep 2015 08:28:00 -0400

> Is it possible to not be patched ?

Traffic from that IP range should have the behavior Eric is referring
to (just HYSTART_DELAY, no HYSTART_ACK_TRAIN).

neal

---

From: Veaceslav ROMAN <[email-masked]>
Date: Thu, 3 Sep 2015 19:10:03 +0000

> Great!

Excuse my ignorance but I don't know the rules of this exchange. Can I send a couple of traces as mail attachments or there is a place for upload, or I can use a public upload ? Each trace file is ~2.2 MB and contain 1 SD and 1 HD download trace. 

> To be useful, the traces would need to be from a kernel that is version 4.0 or newer, as the latest CUBIC changes were in v4.0. You would also need to disable the ACK train detection with:

>  https://lwn.net/Articles/565421/

 It will take a little to setup 4.0, none in my team has an experience with it yet, they are having doubts on hardware and proposes either to buy a new server or to virtualize existing under VMware, even though I do prefer to avoid it, not to be have an unknown layer in the middle.

neal

Veaceslav Roman

-----Original Message-----
From: Neal Cardwell [mailto:[email-masked]] 
Sent: Wednesday, 02 September 2015 15:26
To: Veaceslav ROMAN
Cc: Ingemar Johansson S; Piers O'Hanlon; [email-masked]; Sangtae Ha; Eric Dumazet; [email-masked]
Subject: Re: [e2e] TCP HyStart patch deployment

Great!

> For the server side traces, can you suggest the minimum Linux kernel for which it will make sense to analyze traces?

To be useful, the traces would need to be from a kernel that is version 4.0 or newer, as the latest CUBIC changes were in v4.0. You would also need to disable the ACK train detection with:

neal

> Cc: Ingemar Johansson S; Piers O'Hanlon; [email-masked]; Sangtae Ha; 
> Eric Dumazet; [email-masked]

---

From: Neal Cardwell <[email-masked]>
Date: Thu, 3 Sep 2015 15:19:38 -0400

On Thu, Sep 3, 2015 at 3:10 PM, Veaceslav ROMAN
<[email-masked]> wrote:
> Excuse my ignorance but I don't know the rules of this exchange.
> Can I send a couple of traces as mail attachments or there is a
> place for upload, or I can use a public upload ? Each trace file is
> ~2.2 MB and contain 1 SD and 1 HD download trace.

My guess is that for an file of that size you'd be best off with
posting it on a publicly accessible web URL somewhere under your
control.

>  It will take a little to setup 4.0, none in my team has an experience with it yet,

OK. What kernel version are you running? I guess you could just patch
your hystart_update() in tcp_cubic.c to look like v4.0, disable
Hystart ACK train detection, and enable fq. That should capture most
of what would be interesting.

neal

---

From: Eric Dumazet <[email-masked]>
Date: Thu, 03 Sep 2015 14:14:02 -0700

On Thu, 2015-09-03 at 15:19 -0400, Neal Cardwell wrote:
> tps://lwn.net/Articles/565421/
> >

AFAIK no kernel patch/build is even required.

echo 2 >/sys/module/tcp_cubic/parameters/hystart_detect

And that should be good ;)

For the fq pacing stuff, you could try the following,
even if it is not 'perfect' in the sense you wont have multiqueue
support if your NIC is capable of.

tc qdisc replace dev eth0 root fq

---

From: Veaceslav ROMAN <[email-masked]>
Date: Thu, 3 Sep 2015 21:22:03 +0000

Thank you,
Will follow your suggestions.
I also guess we'll better switch off GSO/TSO ?

Veaceslav Roman

-----Original Message-----
From: Eric Dumazet [mailto:[email-masked]] 
Sent: Friday, 04 September 2015 00:14
To: Neal Cardwell
Cc: Veaceslav ROMAN; [email-masked]; Piers O'Hanlon; Sangtae Ha; Ingemar Johansson S; Eric Dumazet; [email-masked]
Subject: Re: [tcpm] [e2e] TCP HyStart patch deployment

For the fq pacing stuff, you could try the following, even if it is not 'perfect' in the sense you wont have multiqueue support if your NIC is capable of.

---

From: Eric Dumazet <[email-masked]>
Date: Thu, 03 Sep 2015 14:28:01 -0700

> I also guess we'll better switch off GSO/TSO ?

Unless you absolutely want bad performance, you should not.

If it makes you feel better, know that Google never disabled TSO/GSO or
GRO if that matters.

---

From: Veaceslav ROMAN <[email-masked]>
Date: Thu, 3 Sep 2015 22:15:22 +0000

Neal, I've sent 3 pcaps, to you only, via Google Drive, to check whether you can get them, never used the feature before. Please, let me know if you've got them.
In total we have ~900 files, but let's see if what I've sent is usable.
Also attached a few graphical representation, these are not intended to be exact, just to show approximately the dynamic.

Few words about the testing configuration:
	Device Samsung Galaxy Note4LTE CAT6 (SM-N910F) LTE / HSPA+ DC / HSUPA 5.76 
             Radio Network LTE 2600/1800/800 + Carrier Aggregation, up to 200 Mbps capable
             Distance between the Core Network (EPC) and Base Station - 0 - 250 km
             Youtube server in Bucharest (Romania)
             Distance between Core Network and Youtube server ~460 km (by road Chisinau (Moldova) - Bucharest(Romania)
             Internet connectivity 10 Gbps
             Core network switching(EPC) 10 Gbps
             Backhaul transport (between the core network and regional PoP) 1 Gbps, full IP, MPLS
             Last mile transport (between regional PoP and radio base station eNodeB), full IP, MPLS,  300 Mbps

              Each pcap shows 1 SD (~2 MB) and 1 HD (~8MB) video downloads.
              
Why suspect Hystart ? In most typical scenario, corresponding to a file " 2015-07-01_12-31-46-589_00009444.TCP_IP.pcap", an aggressive grow of the bytes-in-flight up to 50 KB then very slow, almost linear growth, up to 200 KB (SD download) or 400 KB (HD download), no drops but in the same time the RTT is not growing (i.e. no buffering). Accordingly speed grows slowly from 10 to 30-40 Mbps and then... the download is finished.
Out of 1800 downloads (900 SD + 900 HD) there are less than 400 which have any drop during the download. Unfortunately the average speed is much less than one can get from the network should the slow start not finishing too early.

Veaceslav Roman

-----Original Message-----
From: Neal Cardwell [mailto:[email-masked]] 
Sent: Thursday, 03 September 2015 22:20
To: Veaceslav ROMAN
Cc: Ingemar Johansson S; Piers O'Hanlon; [email-masked]; Sangtae Ha; Eric Dumazet; [email-masked]
Subject: Re: [e2e] TCP HyStart patch deployment

On Thu, Sep 3, 2015 at 3:10 PM, Veaceslav ROMAN <[email-masked]> wrote:
> Can I send a couple of traces as mail attachments or there is a place 
> for upload, or I can use a public upload ? Each trace file is

My guess is that for an file of that size you'd be best off with posting it on a publicly accessible web URL somewhere under your control.

>  It will take a little to setup 4.0, none in my team has an experience 

OK. What kernel version are you running? I guess you could just patch your hystart_update() in tcp_cubic.c to look like v4.0, disable Hystart ACK train detection, and enable fq. That should capture most of what would be interesting.

neal

---

From: Neal Cardwell <[email-masked]>
Date: Thu, 3 Sep 2015 19:45:14 -0400

> And that should be good ;)

I was just suggesting that he copy the v4.0 hystart_update() in
tcp_cubic.c to make sure he picked up your relatively recent change to
Hystart, from v3.19:

  42eef7a tcp_cubic: refine Hystart delay threshold

since that was a significant change to the behavior of Hystart, to
make it less sensitive to delays.

neal

---

From: Neal Cardwell <[email-masked]>
Date: Fri, 4 Sep 2015 11:32:07 -0400

Hi Veaceslav,

I agree that in your LTE traces it looks like CUBIC Hystart is exiting
slow start too early.

SInce you seem to have a nice LTE testbed, would you be able to do
some experiments to find a set of parameters for Hystart that work
better for your LTE environment? For example, you might try the two
variations I suggested earlier in the thread:

                        if (ca->curr_rtt > ca->delay_min +
                            HYSTART_DELAY_THRESH(ca->delay_min >> 2)) {
or
                        if (ca->curr_rtt > ca->delay_min +
                            HYSTART_DELAY_THRESH(ca->delay_min >> 1)) {

Do any of those give better results for your tests?

neal

---

From: Eric Dumazet <[email-masked]>
Date: Fri, 04 Sep 2015 09:48:17 -0700

> Do any of those give better results for your tests?

Also, hystart is fooled by too many ACK received in short period.

This problem would be solved if GRO was used at receiver, as less ACK
would be sent.

Presumably receiver is not a linux TCP stack ?

Maybe we should add a logic in hystart_update() to take one ACK per ms.

05:32:24.134172 IP 94.243.104.156.32924 > 195.95.178.204.80: Flags [S], seq 4294423215, win 65535, options [mss 1460,sackOK,TS val 1112351 ecr 0,nop,wscale 8], length 0
05:32:24.161986 IP 195.95.178.204.80 > 94.243.104.156.32924: Flags [S.], seq 1198877721, ack 4294423216, win 28960, options [mss 1416,sackOK,TS val 2808808634 ecr 1112351,nop,wscale 7], length 0
05:32:24.162108 IP 94.243.104.156.32924 > 195.95.178.204.80: Flags [.], ack 1, win 343, options [nop,nop,TS val 1112354 ecr 2808808634], length 0
05:32:24.163072 IP 94.243.104.156.32924 > 195.95.178.204.80: Flags [P.], seq 1:658, ack 1, win 343, options [nop,nop,TS val 1112354 ecr 2808808634], length 657
05:32:24.203984 IP 195.95.178.204.80 > 94.243.104.156.32924: Flags [.], ack 658, win 237, options [nop,nop,TS val 2808808675 ecr 1112354], length 0
05:32:24.210275 IP 195.95.178.204.80 > 94.243.104.156.32924: Flags [P.], seq 1:387, ack 658, win 237, options [nop,nop,TS val 2808808677 ecr 1112354], length 386
05:32:24.210315 IP 195.95.178.204.80 > 94.243.104.156.32924: Flags [.], seq 387:1791, ack 658, win 237, options [nop,nop,TS val 2808808677 ecr 1112354], length 1404
05:32:24.210332 IP 195.95.178.204.80 > 94.243.104.156.32924: Flags [.], seq 1791:3195, ack 658, win 237, options [nop,nop,TS val 2808808677 ecr 1112354], length 1404
05:32:24.210347 IP 195.95.178.204.80 > 94.243.104.156.32924: Flags [.], seq 3195:4599, ack 658, win 237, options [nop,nop,TS val 2808808677 ecr 1112354], length 1404
05:32:24.210350 IP 94.243.104.156.32924 > 195.95.178.204.80: Flags [.], ack 387, win 347, options [nop,nop,TS val 1112359 ecr 2808808677], length 0
05:32:24.210361 IP 195.95.178.204.80 > 94.243.104.156.32924: Flags [.], seq 4599:6003, ack 658, win 237, options [nop,nop,TS val 2808808677 ecr 1112354], length 1404
05:32:24.210374 IP 195.95.178.204.80 > 94.243.104.156.32924: Flags [.], seq 6003:7407, ack 658, win 237, options [nop,nop,TS val 2808808677 ecr 1112354], length 1404
05:32:24.210389 IP 195.95.178.204.80 > 94.243.104.156.32924: Flags [.], seq 7407:8811, ack 658, win 237, options [nop,nop,TS val 2808808677 ecr 1112354], length 1404
05:32:24.210402 IP 195.95.178.204.80 > 94.243.104.156.32924: Flags [.], seq 8811:10215, ack 658, win 237, options [nop,nop,TS val 2808808677 ecr 1112354], length 1404
05:32:24.210416 IP 195.95.178.204.80 > 94.243.104.156.32924: Flags [.], seq 10215:11619, ack 658, win 237, options [nop,nop,TS val 2808808677 ecr 1112354], length 1404
05:32:24.210418 IP 94.243.104.156.32924 > 195.95.178.204.80: Flags [.], ack 1791, win 358, options [nop,nop,TS val 1112359 ecr 2808808677], length 0
05:32:24.210431 IP 195.95.178.204.80 > 94.243.104.156.32924: Flags [.], seq 11619:13023, ack 658, win 237, options [nop,nop,TS val 2808808677 ecr 1112354], length 1404
05:32:24.210455 IP 94.243.104.156.32924 > 195.95.178.204.80: Flags [.], ack 3195, win 369, options [nop,nop,TS val 1112359 ecr 2808808677], length 0
05:32:24.210685 IP 94.243.104.156.32924 > 195.95.178.204.80: Flags [.], ack 4599, win 380, options [nop,nop,TS val 1112359 ecr 2808808677], length 0
05:32:24.210731 IP 94.243.104.156.32924 > 195.95.178.204.80: Flags [.], ack 6003, win 391, options [nop,nop,TS val 1112359 ecr 2808808677], length 0
05:32:24.210937 IP 94.243.104.156.32924 > 195.95.178.204.80: Flags [.], ack 7407, win 402, options [nop,nop,TS val 1112359 ecr 2808808677], length 0
05:32:24.211078 IP 94.243.104.156.32924 > 195.95.178.204.80: Flags [.], ack 8811, win 413, options [nop,nop,TS val 1112359 ecr 2808808677], length 0
05:32:24.211228 IP 94.243.104.156.32924 > 195.95.178.204.80: Flags [.], ack 10215, win 424, options [nop,nop,TS val 1112359 ecr 2808808677], length 0
05:32:24.211377 IP 94.243.104.156.32924 > 195.95.178.204.80: Flags [.], ack 11619, win 435, options [nop,nop,TS val 1112359 ecr 2808808677], length 0
05:32:24.211544 IP 94.243.104.156.32924 > 195.95.178.204.80: Flags [.], ack 13023, win 446, options [nop,nop,TS val 1112359 ecr 2808808677], length 0
05:32:24.237999 IP 195.95.178.204.80 > 94.243.104.156.32924: Flags [.], seq 13023:14427, ack 658, win 237, options [nop,nop,TS val 2808808709 ecr 1112359], length 1404
05:32:24.238053 IP 195.95.178.204.80 > 94.243.104.156.32924: Flags [.], seq 14427:15831, ack 658, win 237, options [nop,nop,TS val 2808808709 ecr 1112359], length 1404
05:32:24.238090 IP 94.243.104.156.32924 > 195.95.178.204.80: Flags [.], ack 14427, win 457, options [nop,nop,TS val 1112362 ecr 2808808709], length 0
05:32:24.238164 IP 94.243.104.156.32924 > 195.95.178.204.80: Flags [.], ack 15831, win 468, options [nop,nop,TS val 1112362 ecr 2808808709], length 0
05:32:24.244024 IP 195.95.178.204.80 > 94.243.104.156.32924: Flags [.], seq 15831:17235, ack 658, win 237, options [nop,nop,TS val 2808808713 ecr 1112359], length 1404
05:32:24.244063 IP 195.95.178.204.80 > 94.243.104.156.32924: Flags [.], seq 17235:18639, ack 658, win 237, options [nop,nop,TS val 2808808713 ecr 1112359], length 1404

---

From: Veaceslav ROMAN <[email-masked]>
Date: Fri, 4 Sep 2015 16:58:54 +0000

Unfortunately is not my testbed. It is an external independent supplier who run measurement campaign twice a year. 
We've already installed kernel 4.1 and by the middle of next week we shall be able to take regularly traces and we will experiment with all proposed variations.

Veaceslav Roman

-----Original Message-----
From: Neal Cardwell [mailto:[email-masked]] 
Sent: Friday, 04 September 2015 18:32
To: Eric Dumazet
Cc: Veaceslav ROMAN; [email-masked]; Piers O'Hanlon; Sangtae Ha; Ingemar Johansson S; Eric Dumazet; [email-masked]
Subject: Re: [tcpm] [e2e] TCP HyStart patch deployment

Hi Veaceslav,

I agree that in your LTE traces it looks like CUBIC Hystart is exiting slow start too early.

SInce you seem to have a nice LTE testbed, would you be able to do some experiments to find a set of parameters for Hystart that work better for your LTE environment? For example, you might try the two variations I suggested earlier in the thread:

                        if (ca->curr_rtt > ca->delay_min +
                            HYSTART_DELAY_THRESH(ca->delay_min >> 2)) { or
                        if (ca->curr_rtt > ca->delay_min +
                            HYSTART_DELAY_THRESH(ca->delay_min >> 1)) {

neal

---

From: Eric Dumazet <[email-masked]>
Date: Fri, 04 Sep 2015 11:21:52 -0700

> Maybe we should add a logic in hystart_update() to take one ACK per ms.

BTW, the server has following SNMP stats :

$ nstat -a | egrep "Hystart|PassiveOpens"
TcpPassiveOpens                 308476294          0.0
TcpExtTCPHystartDelayDetect     31204296           0.0
TcpExtTCPHystartDelayCwnd       5549344673         0.0

Meaning that Hystart delay detection exits slow start for ~10% of flows,
with an average cwnd = 177.84

---

From: Veaceslav ROMAN <[email-masked]>
Date: Fri, 4 Sep 2015 22:09:35 +0000

If we look at the first train: all packets received in less than 1 ms. Probably this is only an appearance as LTE transmits in, so called transmission time interval (TTI) of 1 ms, and what we see here is that all 10 packets of initial window fitted in 1 ms, and, when decoded, were presented to the TCP/IP layer (and pcap) all at once. BTW, 8 packets of 13022 bytes in 1 ms means instantaneous speed of 104 Mbps, good radio conditions. TCP generates 10 ACK train of the duration of 1.2 ms. Will it be a Fast Ethernet possibly we would consider this normal, isn't it ?
I looked at how server reply to these 10 ACKs. There are 3 gaps of 4, 2 and 5 ms in the reply train and the total train of 18 (?, it should be 20) packets reaches a duration of 14 ms. AFAIK LTE may introduce gaps in ACK train due to uplink scheduling mechanism. 
May be these gaps trigger Hystart early exit ?

Veaceslav Roman

-----Original Message-----
From: Eric Dumazet [mailto:[email-masked]] 
Sent: Friday, 04 September 2015 19:48
To: Neal Cardwell
Cc: Veaceslav ROMAN; [email-masked]; Piers O'Hanlon; Sangtae Ha; Ingemar Johansson S; Eric Dumazet; [email-masked]
Subject: Re: [tcpm] [e2e] TCP HyStart patch deployment

This problem would be solved if GRO was used at receiver, as less ACK would be sent.

---

From: Veaceslav ROMAN <[email-masked]>
Date: Thu, 1 Oct 2015 00:24:40 +0000

Finally can share some results of Hystart interaction with LTE.
 Few words about the testing configuration:
 	Device Samsung Galaxy Note4LTE , up to 150 Mbps capable 
             Radio Network LTE 2600/1800
             Distance between the Core Network (EPC) and Base Station - 10 km
             HTTP Server connected directly to the Core Network 1 Gbps
             Core network switching(EPC) 10 Gbps
             Backhaul transport  1 Gbps, full IP, MPLS
             Last mile transport full IP, MPLS,  300 Mbps

            Server: Linux k40srv 4.1.0-040100-generic #201507030940 SMP Fri Jul 3 09:41:47 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
                        Apache/2.4.10 (Ubuntu), built:   Jul 24 2015 17:25:18 

           Hystart_detect: 2 (HYSTART_DELAY only), as suggested.
           net.ipv4.tcp_no_metrics_save = 1 to avoid tcp metrics caching interference 

	Tests types: 
		1. Download 3 MB file, interchanging Hystart ON and OFF, in conditions of Low Bandwidth (25-35 Mbps, poor radio) and High Bandwidth (100-120 Mbps, good radio).
		2. Download 10 MB file, interchanging Hystart ON and OFF, in conditions of Low Bandwidth (25-35 Mbps, poor radio) and High Bandwidth (100-120 Mbps, good radio).

 	100 Download tests per type. 
             Download duration and bandwidth measured at the client side as shown by curl from the first received data packet till the end of session (this exclude DNS, 3 way handshake, http GET and its ACK). 
             Hystart exit cwnd extracted from nstat (thank you Eric to pointing out to it).      

Overall results:
	Download 3MB, Low Bandwidths: 
		Hystart average download time: 1.41 s
 		NoHystart average download time: 1.36 s              

	Download 3MB, High Bandwidths: 
		Hystart average download time: 0.65 s
 		NoHystart average download time: 0.37  s              
		
	Download 10MB, Low Bandwidths: 
		Hystart average download time: 2.40 s
 		NoHystart average download time: 2.12 s              

	Download 10MB, High Bandwidths: 
		Hystart average download time: 1.42 s
 		NoHystart average download time: 0.89  s     

These results show that Hystart has no significant impact in conditions of low available bandwidth, but at high available the decrease of performance can reach 60-70 %.
With the development of the LTE-A which uses larger spectrum to reach 300 Mbps and more the impact of Hystart will become even more visible.       

I do share with Eric and Neal some files described below and also can provide links to anyone interested.

Summary of these tests results are gathered in the following Excel files (names, I believe, are self explanatory): Test_3M_low_BW.xlsx, Test_3M_high_BW.xlsx, Test_10M_low_BW.xlsx, Test_10M_high_BW.xlsx.
Similarly tcpdump traces are called Test_3M_low_BW.pcap, Test_3M_high_BW. pcap, Test_10M_low_BW. pcap, Test_10M_high_BW. pcap (able to collect only server side traces, but they are self sufficient).
For the 10M downloads there are few missing traces as there trace files were recycled. In traces are also ssh sessions, please, ignore them, they were used to put hystart ON and OFF on server and extract nstat results before and after each download.

Excel files have 2 tabs: curl_sum and Summary.
In the curl_sum tab the stat is cumulated in chronological order of tests.
In the Summary_tab  the Hystart and NoHystart tests are shown side by side (as said, Download 1 was with Hystart, Download 2 without, Download 3 with Hystart, Download 4 without, etc.). It also include graphs of Transfer time and a graph which shows the distribution in % of different Hystart exit window DelayCWND.

Fields:
comments	- test case
Transf_Time	- calculated from curl output the duration between first data packet and end of transfer
Transf_speed	- calculated from curl output the ratio of file size and duration between first data packet and end of transfer
Hystart	D_Delay -  hystart exit occurrence (difference between TcpExtTCPHystartDelayDetect before and after download)	
D_DelayCwnd	  - Hystart exit cwnd for this download (difference of TcpExtTCPHystartDelayCwnd before and after the download)
Date_On_Server - time just before the download start, to help navigate the pcap

Additionally, in the file Test_10M_low_BW.xlsx in the tab curl_sum are included for each download tcp.stream extract from pcap of the first 100 tcp.analysis.ack_rtt which may help in understanding of the hystart behavior (fields ack_rtt_1 to ack_rtt_100).
In this file there are also fields URI_frame, URI_time, URI_time_relative,	tcp_stream.
Then there are calculated fields: minRTT as a minimum of all 100 ACK, r1_5, r2_8, r3_8,	r4_8 which simulate Hystart calculations of the round curr_rtt and r1 min, r2 min, r3 min, r4 min represent minRTT of the whole round.
Please, correct me if I am wrong, but in my understanding for each ACK the hystart_update is called before the hystart_reset, and as a result the computation of cur_rtt of the 8 packets of the round starts with the 2-nd packet of the round, not the first one. 
 I've use r1_5 as a min(ack_rtt_7...ack_rtt_11), due to hystart_low_window=16, r2_8 as a min(ack_rtt_12 ... ack_rtt_19), r3_8 as a min(ack_rtt_32 ... ack_rtt_39) and ), r4_8 as a min(ack_rtt_72 ... ack_rtt_79).
To understand the impact of this there are also r1 HOL, r2 HOL, r3 HOL, r4 HOL (Head Of Line) which compute the same with first packet in the round included in the relevant round:  r1_5 as a min(ack_rtt_6...ack_rtt_10), r2_8 as a min(ack_rtt_11 ... ack_rtt_18), r3_8 as a min(ack_rtt_31 ... ack_rtt_38) and ), r4_8 as a min(ack_rtt_71 ... ack_rtt_78).

If my understanding of Hystart is correct and, assuming in each train the first packet has the lowest RTT which, most probably is valid in all type of networks, then the exit from hystart in this implementation may happen at cwnd 29, 49, 89, 169, etc. (given the IW of 10).
These tests show that this is the case, however there also intermediate values. However the cwnd of 29 shall be the lowest possible value and this is the case in these tests. Unfortunately, in LTE the first packet in the train has chances to always have ~6 ms lower than the following ones and, because it is not counted in the 2-nd train, there is quite high percentage of exit at 29 (14-36% in these tests), even though later on cubic increases the cwnd without losses up to many hundreds. For me this is too early exit from slow start.
As it can be seen from comparison of the simulated calculation, inclusion of the first packet in the round curr_rtt calculation would eliminate the cwnd 29 and shift to cwnd 49 or 89 or higher.

But there are also intermediate values, e.g. 69. Looking through traces one may see that this is due to "compressed ACK" which receives a tightly packed train of many ACK so that in traces there are not data packets sent in turn by server. Looks to me, because hysrart_reset sets the end_seq to snd_next and the server did not managed to send yet packets in reply to this high speed ACK train the snd_next points to much lower value than would normally happen and this actually destroys the correct identification of the borders of the round, the round fails somewhere in the middle of the train not at its border, this makes very likely increased delay due to queueing and earlier exit from slow_start and intermediate values of exit cwnd where they are not expected.

And last but not least: there are gaps (4-7 ms) in ACK trains. On the perfectly sent IW of 10 data packets train the typical for LTE will be to receive 1 or 2 ACK, then a gap of 4-6 ms then another couple of ACK, then another gap, then 5 ACK compressed together in a fraction of microsecond. Of cause, the next train from the server will contain the same gaps. Very quickly, e.g., the Head of Line (HOL) of the 3-d train will follow immediately the tail of the 2-nd train which will lead to queueing in the end router (actually the radio base station, much earlier than BDP reached), but this will lead to increase of the train RTT and too yearly exit from slow start. Why too early exit ? Because, should server continue increasing the speed this would reduce quicker the gaps both in downlink and uplink, the preice for this being the increase of the RTT (I guess double) against what one would normally see in the end-to-end Ethernet networks.

I do like ideas of Hystart, but I  don't know how could it be possibly reconciled with my LTE problems.  With the increase of LTE speeds this early exit from slow start will become even more of the problem.
For LTE, not sure how good and generally acceptable, solution would be, possibly, to increase the HYSTART_DELAY_MIN to 8 ms, which, at least in my case, would diminish the earlier exit significantly, if I believe in these tests results.
Another solution, which I don't like very much, but as I know some operators use, would be to insert a kind of TCP accelerator between the Internet and the mobile network which will intercept all TCP and would send it to mobile without Hystart but this would kill any end-to-end principle and this will not be the Internet anymore.

If you have time and never look closely to LTE technology I do a quick summary bellow.

LTE, TCP self-clocking and ACK compression.

Spent some time to understand the basics of what could be the reason for all those timing effects due to LTE.
	First, unlike wire based transport technologies, LTE (but also UMTS and future 5G) sends data, both uplink and downlink in predefined timeslots called Transmission Time Interval (TTI). In UMTS this is 2ms, in LTE is 1 ms and it says that in 5G it will be 0.5 ms.
These TTI have exact borders and tightly follow each other and require exact synchronization between the mobile device and Radio Base Station (called eNodeB in LTE). To cope with varying radio conditions the sending data bandwidth is modified in such a way that to keep a constant probability of data loss of 10%. Bad radio - less bandwidth, good radio - more bandwidth. The bandwidth is modified by packing less or more bits in a TTI of 1 ms in LTE. But TTI remain unchanged of exactly 1 ms. That means that, at the TCP/IP layer the precision of the timing information from device to the network is 1 ms. In the case of a bandwidth of 1 Mbps there will be 83 IP packets per second or one packet every 12 ms. In this case, possibly the return of one ACK every 12 ms (every 12 TTI) would be more or less sufficient to keep at a good level the TCP self-clocking mechanism. But, when talking about 100 Mbps (or 300 Mbps) that means one packet (and one ACK) every 0.12 ms and there is no way to provide it with a TTI of 1 ms. This is the reason for ACK compression in LTE and the future 5G which promises more than 1Gbps with its 0.5 ms TTI will be in a big conflict with TCP self-clocking.
	But the problem do not stops here. There is a significantly different mechanisms of transmission in the downlink (from eNodeB to mobile device) and in uplink (from mobile device to the eNodeB). The difference come from the fact that the scheduler for both downlink and in uplink is located in the eNodeB. As of my understanding the purpose of this is to simplify to the maximum the device and save its battery. As a result of this design, the eNodeB can send in each TTI toward the mobile device whenever data exist in its buffer (eNodeB is a kind of IP wireless router). But for mobile device, in order to send the data in uplink, after a pause, it has first to ask the network to schedule it. And mobile has for this request (after a pause of sending) just one bit, actually is not even a data bit, is a special radio signal which tells "Hey, I have some data in the buffer to send". Lets look at initial train of IW of 10 packets. In good radio conditions the eNodeB will fit all 10 packets in 1 TTI of 1 ms and send them all to the device. Device will unpack them and generate 10 ACK and then tell to the eNodeB "Hey, I have some data in the buffer to send". The eNodeB has not knowledge how much data the mobile has to send, therefor will allocate some minimum capacity: "Well, I give you a grant to send up to 200 Bytes and you must send them exactly after 4 ms you get this notification". These 4 ms are allowance for mobile device to prepare data for sending which is a very complicated process, these 4 ms is a must delay between the grant and the transmission. If mobile has in its buffer 10 ACK to send it will send only 2 ACK and will complement them with, so called, buffer status report: "I do have still 480 bytes to send". Now network has more information and allocate as requested: "Well, I give you a grant to send up to 200 Bytes and you must send them exactly after 4 ms you get this notification". But, in the mean time some other information of higher priority may appear in the device (e.g. signaling) and the device will send only 6 out of 8 remaining ACK together with signaling and another Buffer Status Report: "Hey, I do have another 120 bytes to send". And again "Well, I give you a grant to send up to 120 Bytes and you must send them exactly after 4 ms you get this notification". As a result the server will receive 2 ACK, then after a gap of 6 ms another 6 ACK, then, after another 6 ms, the remaining 2 ACK. With the scheduler in the eNodeB these gaps are inevitable. And one more think: mobile pack together those 6 ACK in 1 TTI of 1 ms and send them, then the eNodeB unpack them and there is no way to recover the spacing of 0.12 ms between them. eNodeB simply send them to the server back to back at the speed of its data card which is typically 1 Gbps and we face the ACK compression. After the last 2 ACK were sent there is nothing else to send, so there is no Buffer Status Report, therefore, after packets of the next train arrive and ACK are generated everything starts again with one bit of information "Hey, I have some data in the buffer to send". 
That's why there are gaps and ACK compression, loss, or more exactly, lack of any timing information which would be useful for the TCP self-clocking.  
   
Thank you and sorry for taking your time.

Veaceslav Roman

-----Original Message-----
From: Veaceslav ROMAN 
Sent: Saturday, 05 September 2015 01:10
To: 'Eric Dumazet'; Neal Cardwell
Cc: [email-masked]; Piers O'Hanlon; Sangtae Ha; Ingemar Johansson S; Eric Dumazet; [email-masked]
Subject: RE: [tcpm] [e2e] TCP HyStart patch deployment

Veaceslav Roman

---

From: Ingemar Johansson S <[email-masked]>
Date: Thu, 1 Oct 2015 05:58:55 +0000

Hi

Thanks for the detailed report. What seems unclear to me is the value of HYSTART_DELAY_MIN. You mention 8ms but when I look into the 4.2 kernel code I get the feeling that it is 32ms ( #define HYSTART_DELAY_MIN       (4U<<3)   ). Is this a mistake from my side  ?
I have run LTE simulations and tried to wring this HyStart issue inside out and upside down but sofar it has been very difficult to make HyStart exit prematurely but then of course I have run with HYSTART_DELAY_MIN = 32ms. 

As regards to the ACK-compression problem, yes, ACK compression easily occur in LTE, even in simulations. What I have seen is that packet pacing is a very efficient remedy, for instance the packet implemented in QUIC seems to solve ACK compression issues very well. 

It is interesting what you say that ACK trains are so much split up, I could understand that they are split up in two parts (initial grant + additional after received BSR).  

/Ingemar

> From: Veaceslav ROMAN [mailto:[email-masked]]
> Sent: den 1 oktober 2015 02:25
> To: Eric Dumazet; Neal Cardwell
>             Server: Linux k40srv 4.1.0-040100-generic #201507030940 SMP Fri Jul 3
> 09:41:47 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
> 		1. Download 3 MB file, interchanging Hystart ON
> and OFF, in conditions of Low Bandwidth (25-35 Mbps, poor radio) and High
> Bandwidth (100-120 Mbps, good radio).
> 		2. Download 10 MB file, interchanging Hystart
> ON and OFF, in conditions of Low Bandwidth (25-35 Mbps, poor radio) and
> High Bandwidth (100-120 Mbps, good radio).
> 
>              Download duration and bandwidth measured at the client side as
> shown by curl from the first received data packet till the end of session (this
> exclude DNS, 3 way handshake, http GET and its ACK).
> These results show that Hystart has no significant impact in conditions of low
> available bandwidth, but at high available the decrease of performance can
> reach 60-70 %.
> With the development of the LTE-A which uses larger spectrum to reach 300
> Mbps and more the impact of Hystart will become even more visible.
> 
> Summary of these tests results are gathered in the following Excel files
> (names, I believe, are self explanatory): Test_3M_low_BW.xlsx,
> Test_3M_high_BW.xlsx, Test_10M_low_BW.xlsx, Test_10M_high_BW.xlsx.
> Similarly tcpdump traces are called Test_3M_low_BW.pcap,
> Test_3M_high_BW. pcap, Test_10M_low_BW. pcap, Test_10M_high_BW.
> pcap (able to collect only server side traces, but they are self sufficient).
> For the 10M downloads there are few missing traces as there trace files were
> recycled. In traces are also ssh sessions, please, ignore them, they were used
> to put hystart ON and OFF on server and extract nstat results before and
> after each download.
> 
> In the Summary_tab  the Hystart and NoHystart tests are shown side by side
> (as said, Download 1 was with Hystart, Download 2 without, Download 3 with
> Hystart, Download 4 without, etc.). It also include graphs of Transfer time and
> a graph which shows the distribution in % of different Hystart exit window
> DelayCWND.
> 
> Fields:
> comments	- test case
> Transf_speed	- calculated from curl output the ratio of file size and duration
> between first data packet and end of transfer
> Hystart	D_Delay -  hystart exit occurrence (difference between
> TcpExtTCPHystartDelayDetect before and after download)
> D_DelayCwnd	  - Hystart exit cwnd for this download
> (difference of TcpExtTCPHystartDelayCwnd before and after the download)
> Additionally, in the file Test_10M_low_BW.xlsx in the tab curl_sum are
> included for each download tcp.stream extract from pcap of the first 100
> tcp.analysis.ack_rtt which may help in understanding of the hystart behavior
> (fields ack_rtt_1 to ack_rtt_100).
> Then there are calculated fields: minRTT as a minimum of all 100 ACK, r1_5,
> r2_8, r3_8,	r4_8 which simulate Hystart calculations of the round curr_rtt
> and r1 min, r2 min, r3 min, r4 min represent minRTT of the whole round.
> Please, correct me if I am wrong, but in my understanding for each ACK the
> hystart_update is called before the hystart_reset, and as a result the
> computation of cur_rtt of the 8 packets of the round starts with the 2-nd
> packet of the round, not the first one.
>  I've use r1_5 as a min(ack_rtt_7...ack_rtt_11), due to
> hystart_low_window=16, r2_8 as a min(ack_rtt_12 ... ack_rtt_19), r3_8 as a
> min(ack_rtt_32 ... ack_rtt_39) and ), r4_8 as a min(ack_rtt_72 ... ack_rtt_79).
> To understand the impact of this there are also r1 HOL, r2 HOL, r3 HOL, r4 HOL
> (Head Of Line) which compute the same with first packet in the round
> included in the relevant round:  r1_5 as a min(ack_rtt_6...ack_rtt_10), r2_8 as
> a min(ack_rtt_11 ... ack_rtt_18), r3_8 as a min(ack_rtt_31 ... ack_rtt_38) and
> ), r4_8 as a min(ack_rtt_71 ... ack_rtt_78).
> 
> If my understanding of Hystart is correct and, assuming in each train the first
> packet has the lowest RTT which, most probably is valid in all type of
> networks, then the exit from hystart in this implementation may happen at
> cwnd 29, 49, 89, 169, etc. (given the IW of 10).
> These tests show that this is the case, however there also intermediate
> values. However the cwnd of 29 shall be the lowest possible value and this is
> the case in these tests. Unfortunately, in LTE the first packet in the train has
> chances to always have ~6 ms lower than the following ones and, because it
> is not counted in the 2-nd train, there is quite high percentage of exit at 29
> (14-36% in these tests), even though later on cubic increases the cwnd
> without losses up to many hundreds. For me this is too early exit from slow
> start.
> As it can be seen from comparison of the simulated calculation, inclusion of
> the first packet in the round curr_rtt calculation would eliminate the cwnd 29
> and shift to cwnd 49 or 89 or higher.
> 
> But there are also intermediate values, e.g. 69. Looking through traces one
> may see that this is due to "compressed ACK" which receives a tightly packed
> train of many ACK so that in traces there are not data packets sent in turn by
> server. Looks to me, because hysrart_reset sets the end_seq to snd_next
> and the server did not managed to send yet packets in reply to this high
> speed ACK train the snd_next points to much lower value than would
> normally happen and this actually destroys the correct identification of the
> borders of the round, the round fails somewhere in the middle of the train
> not at its border, this makes very likely increased delay due to queueing and
> earlier exit from slow_start and intermediate values of exit cwnd where they
> are not expected.
> 
> And last but not least: there are gaps (4-7 ms) in ACK trains. On the perfectly
> sent IW of 10 data packets train the typical for LTE will be to receive 1 or 2
> ACK, then a gap of 4-6 ms then another couple of ACK, then another gap,
> then 5 ACK compressed together in a fraction of microsecond. Of cause, the
> next train from the server will contain the same gaps. Very quickly, e.g., the
> Head of Line (HOL) of the 3-d train will follow immediately the tail of the 2-nd
> train which will lead to queueing in the end router (actually the radio base
> station, much earlier than BDP reached), but this will lead to increase of the
> train RTT and too yearly exit from slow start. Why too early exit ? Because,
> should server continue increasing the speed this would reduce quicker the
> gaps both in downlink and uplink, the preice for this being the increase of the
> RTT (I guess double) against what one would normally see in the end-to-end
> Ethernet networks.
> 
> 
> I do like ideas of Hystart, but I  don't know how could it be possibly reconciled
> with my LTE problems.  With the increase of LTE speeds this early exit from
> slow start will become even more of the problem.
> For LTE, not sure how good and generally acceptable, solution would be,
> possibly, to increase the HYSTART_DELAY_MIN to 8 ms, which, at least in my
> case, would diminish the earlier exit significantly, if I believe in these tests
> results.
> Another solution, which I don't like very much, but as I know some operators
> use, would be to insert a kind of TCP accelerator between the Internet and
> the mobile network which will intercept all TCP and would send it to mobile
> without Hystart but this would kill any end-to-end principle and this will not
> be the Internet anymore.
> 
> 
> 	First, unlike wire based transport technologies, LTE (but also
> UMTS and future 5G) sends data, both uplink and downlink in predefined
> timeslots called Transmission Time Interval (TTI). In UMTS this is 2ms, in LTE is
> 1 ms and it says that in 5G it will be 0.5 ms.
> These TTI have exact borders and tightly follow each other and require exact
> synchronization between the mobile device and Radio Base Station (called
> eNodeB in LTE). To cope with varying radio conditions the sending data
> bandwidth is modified in such a way that to keep a constant probability of
> data loss of 10%. Bad radio - less bandwidth, good radio - more bandwidth.
> The bandwidth is modified by packing less or more bits in a TTI of 1 ms in LTE.
> But TTI remain unchanged of exactly 1 ms. That means that, at the TCP/IP
> layer the precision of the timing information from device to the network is 1
> ms. In the case of a bandwidth of 1 Mbps there will be 83 IP packets per
> second or one packet every 12 ms. In this case, possibly the return of one
> ACK every 12 ms (every 12 TTI) would be more or less sufficient to keep at a
> good level the TCP self-clocking mechanism. But, when talking about 100
> Mbps (or 300 Mbps) that means one packet (and one ACK) every 0.12 ms and
> there is no way to provide it with a TTI of 1 ms. This is the reason for ACK
> compression in LTE and the future 5G which promises more than 1Gbps with
> its 0.5 ms TTI will be in a big conflict with TCP self-clocking.
> 	But the problem do not stops here. There is a significantly
> different mechanisms of transmission in the downlink (from eNodeB to
> mobile device) and in uplink (from mobile device to the eNodeB). The
> difference come from the fact that the scheduler for both downlink and in
> uplink is located in the eNodeB. As of my understanding the purpose of this
> is to simplify to the maximum the device and save its battery. As a result of
> this design, the eNodeB can send in each TTI toward the mobile device
> whenever data exist in its buffer (eNodeB is a kind of IP wireless router). But
> for mobile device, in order to send the data in uplink, after a pause, it has
> first to ask the network to schedule it. And mobile has for this request (after
> a pause of sending) just one bit, actually is not even a data bit, is a special
> radio signal which tells "Hey, I have some data in the buffer to send". Lets
> look at initial train of IW of 10 packets. In good radio conditions the eNodeB
> will fit all 10 packets in 1 TTI of 1 ms and send them all to the device. Device
> will unpack them and generate 10 ACK and then tell to the eNodeB "Hey, I
> have some data in the buffer to send". The eNodeB has not knowledge how
> much data the mobile has to send, therefor will allocate some minimum
> capacity: "Well, I give you a grant to send up to 200 Bytes and you must send
> them exactly after 4 ms you get this notification". These 4 ms are allowance
> for mobile device to prepare data for sending which is a very complicated
> process, these 4 ms is a must delay between the grant and the transmission.
> If mobile has in its buffer 10 ACK to send it will send only 2 ACK and will
> complement them with, so called, buffer status report: "I do have still 480
> bytes to send". Now network has more information and allocate as
> requested: "Well, I give you a grant to send up to 200 Bytes and you must
> send them exactly after 4 ms you get this notification". But, in the mean time
> some other information of higher priority may appear in the device (e.g.
> signaling) and the device will send only 6 out of 8 remaining ACK together
> with signaling and another Buffer Status Report: "Hey, I do have another 120
> bytes to send". And again "Well, I give you a grant to send up to 120 Bytes
> and you must send them exactly after 4 ms you get this notification". As a
> result the server will receive 2 ACK, then after a gap of 6 ms another 6 ACK,
> then, after another 6 ms, the remaining 2 ACK. With the scheduler in the
> eNodeB these gaps are inevitable. And one more think: mobile pack
> together those 6 ACK in 1 TTI of 1 ms and send them, then the eNodeB
> unpack them and there is no way to recover the spacing of 0.12 ms between
> them. eNodeB simply send them to the server back to back at the speed of
> its data card which is typically 1 Gbps and we face the ACK compression. After
> the last 2 ACK were sent there is nothing else to send, so there is no Buffer
> Status Report, therefore, after packets of the next train arrive and ACK are
> generated everything starts again with one bit of information "Hey, I have
> some data in the buffer to send".
> That's why there are gaps and ACK compression, loss, or more exactly, lack of
> any timing information which would be useful for the TCP self-clocking.
> 
> If we look at the first train: all packets received in less than 1 ms. Probably
> this is only an appearance as LTE transmits in, so called transmission time
> interval (TTI) of 1 ms, and what we see here is that all 10 packets of initial
> window fitted in 1 ms, and, when decoded, were presented to the TCP/IP
> layer (and pcap) all at once. BTW, 8 packets of 13022 bytes in 1 ms means
> instantaneous speed of 104 Mbps, good radio conditions. TCP generates 10
> ACK train of the duration of 1.2 ms. Will it be a Fast Ethernet possibly we
> would consider this normal, isn't it ?
> I looked at how server reply to these 10 ACKs. There are 3 gaps of 4, 2 and 5
> ms in the reply train and the total train of 18 (?, it should be 20) packets
> reaches a duration of 14 ms. AFAIK LTE may introduce gaps in ACK train due
> to uplink scheduling mechanism.
> Cc: Veaceslav ROMAN; [email-masked]; Piers O'Hanlon; Sangtae Ha; Ingemar
> Johansson S; Eric Dumazet; [email-masked]
> 05:32:24.134172 IP 94.243.104.156.32924 > 195.95.178.204.80: Flags [S], seq
> 4294423215, win 65535, options [mss 1460,sackOK,TS val 1112351 ecr
> 0,nop,wscale 8], length 0
> 05:32:24.161986 IP 195.95.178.204.80 > 94.243.104.156.32924: Flags [S.], seq
> 1198877721, ack 4294423216, win 28960, options [mss 1416,sackOK,TS val
> 2808808634 ecr 1112351,nop,wscale 7], length 0
> 05:32:24.162108 IP 94.243.104.156.32924 > 195.95.178.204.80: Flags [.], ack 1,
> win 343, options [nop,nop,TS val 1112354 ecr 2808808634], length 0
> 05:32:24.163072 IP 94.243.104.156.32924 > 195.95.178.204.80: Flags [P.], seq
> 1:658, ack 1, win 343, options [nop,nop,TS val 1112354 ecr 2808808634],
> length 657
> 05:32:24.203984 IP 195.95.178.204.80 > 94.243.104.156.32924: Flags [.], ack
> 658, win 237, options [nop,nop,TS val 2808808675 ecr 1112354], length 0
> 05:32:24.210275 IP 195.95.178.204.80 > 94.243.104.156.32924: Flags [P.], seq
> 1:387, ack 658, win 237, options [nop,nop,TS val 2808808677 ecr 1112354],
> length 386
> 05:32:24.210315 IP 195.95.178.204.80 > 94.243.104.156.32924: Flags [.], seq
> 387:1791, ack 658, win 237, options [nop,nop,TS val 2808808677 ecr 1112354],
> length 1404
> 05:32:24.210332 IP 195.95.178.204.80 > 94.243.104.156.32924: Flags [.], seq
> 1791:3195, ack 658, win 237, options [nop,nop,TS val 2808808677 ecr
> 1112354], length 1404
> 05:32:24.210347 IP 195.95.178.204.80 > 94.243.104.156.32924: Flags [.], seq
> 3195:4599, ack 658, win 237, options [nop,nop,TS val 2808808677 ecr
> 1112354], length 1404
> 05:32:24.210350 IP 94.243.104.156.32924 > 195.95.178.204.80: Flags [.], ack
> 387, win 347, options [nop,nop,TS val 1112359 ecr 2808808677], length 0
> 05:32:24.210361 IP 195.95.178.204.80 > 94.243.104.156.32924: Flags [.], seq
> 4599:6003, ack 658, win 237, options [nop,nop,TS val 2808808677 ecr
> 1112354], length 1404
> 05:32:24.210374 IP 195.95.178.204.80 > 94.243.104.156.32924: Flags [.], seq
> 6003:7407, ack 658, win 237, options [nop,nop,TS val 2808808677 ecr
> 1112354], length 1404
> 05:32:24.210389 IP 195.95.178.204.80 > 94.243.104.156.32924: Flags [.], seq
> 7407:8811, ack 658, win 237, options [nop,nop,TS val 2808808677 ecr
> 1112354], length 1404
> 05:32:24.210402 IP 195.95.178.204.80 > 94.243.104.156.32924: Flags [.], seq
> 8811:10215, ack 658, win 237, options [nop,nop,TS val 2808808677 ecr
> 1112354], length 1404
> 05:32:24.210416 IP 195.95.178.204.80 > 94.243.104.156.32924: Flags [.], seq
> 10215:11619, ack 658, win 237, options [nop,nop,TS val 2808808677 ecr
> 1112354], length 1404
> 05:32:24.210418 IP 94.243.104.156.32924 > 195.95.178.204.80: Flags [.], ack
> 1791, win 358, options [nop,nop,TS val 1112359 ecr 2808808677], length 0
> 05:32:24.210431 IP 195.95.178.204.80 > 94.243.104.156.32924: Flags [.], seq
> 11619:13023, ack 658, win 237, options [nop,nop,TS val 2808808677 ecr
> 1112354], length 1404
> 05:32:24.210455 IP 94.243.104.156.32924 > 195.95.178.204.80: Flags [.], ack
> 3195, win 369, options [nop,nop,TS val 1112359 ecr 2808808677], length 0
> 05:32:24.210685 IP 94.243.104.156.32924 > 195.95.178.204.80: Flags [.], ack
> 4599, win 380, options [nop,nop,TS val 1112359 ecr 2808808677], length 0
> 05:32:24.210731 IP 94.243.104.156.32924 > 195.95.178.204.80: Flags [.], ack
> 6003, win 391, options [nop,nop,TS val 1112359 ecr 2808808677], length 0
> 05:32:24.210937 IP 94.243.104.156.32924 > 195.95.178.204.80: Flags [.], ack
> 7407, win 402, options [nop,nop,TS val 1112359 ecr 2808808677], length 0
> 05:32:24.211078 IP 94.243.104.156.32924 > 195.95.178.204.80: Flags [.], ack
> 8811, win 413, options [nop,nop,TS val 1112359 ecr 2808808677], length 0
> 05:32:24.211228 IP 94.243.104.156.32924 > 195.95.178.204.80: Flags [.], ack
> 10215, win 424, options [nop,nop,TS val 1112359 ecr 2808808677], length 0
> 05:32:24.211377 IP 94.243.104.156.32924 > 195.95.178.204.80: Flags [.], ack
> 11619, win 435, options [nop,nop,TS val 1112359 ecr 2808808677], length 0
> 05:32:24.211544 IP 94.243.104.156.32924 > 195.95.178.204.80: Flags [.], ack
> 13023, win 446, options [nop,nop,TS val 1112359 ecr 2808808677], length 0
> 05:32:24.237999 IP 195.95.178.204.80 > 94.243.104.156.32924: Flags [.], seq
> 13023:14427, ack 658, win 237, options [nop,nop,TS val 2808808709 ecr
> 1112359], length 1404
> 05:32:24.238053 IP 195.95.178.204.80 > 94.243.104.156.32924: Flags [.], seq
> 14427:15831, ack 658, win 237, options [nop,nop,TS val 2808808709 ecr
> 1112359], length 1404
> 05:32:24.238090 IP 94.243.104.156.32924 > 195.95.178.204.80: Flags [.], ack
> 14427, win 457, options [nop,nop,TS val 1112362 ecr 2808808709], length 0
> 05:32:24.238164 IP 94.243.104.156.32924 > 195.95.178.204.80: Flags [.], ack
> 15831, win 468, options [nop,nop,TS val 1112362 ecr 2808808709], length 0
> 05:32:24.244024 IP 195.95.178.204.80 > 94.243.104.156.32924: Flags [.], seq
> 15831:17235, ack 658, win 237, options [nop,nop,TS val 2808808713 ecr
> 1112359], length 1404
> 05:32:24.244063 IP 195.95.178.204.80 > 94.243.104.156.32924: Flags [.], seq
> 17235:18639, ack 658, win 237, options [nop,nop,TS val 2808808713 ecr
> 1112359], length 1404
>