Subject: Re: [tcpm] Vicious or Virtuous circle? Adapting reordering window to reordering degree

From: [email-masked]
Date: Wed, 21 Mar 2018 13:17:07 +0100

>>
>> One datapoint I would like to see would be a comparison of
>> the receiving tcp stack cpu cost, comparing persistent, heavy
>> reordering at around 1/2 RTT at the same sender bandwidth
>> as a "proper" in-order delivered datastream...
>That's a great point: extreme reordering (e.g. packet spray) today
> can cause severe CPU impact beyond protocol reactions
>
> --> extreme reordering would effectively disable GRO
> --> TCP receiver is forced to process packets in MTU unit
> --> TCP receiver sends far more ACKs (GRO produces stretched ACKs)

But would it need to? At high speeds, adjusting the GRO/LRO driver to coalesce more
packets before sending them up for processing might be a easy mitigation, but would add a bit of latency (10..100 usec). However, the reduction of CPU overhead as you described would probably be worth it.

> --> TCP sender is overloaded with ACKs acking MSS-worth of bytes a time
>
> On high speed networks (>1Gbps) with Linux, the CPU impact is far
> more severe than protocol reactions (e.g. RACK or not) that bottlenecks
> the throughput
>
> In general it'd be good to maintain TSO/GRO efficiency when
> constructing flowlets, IMO. There is a nice work in SIGCOMM 2015
> https://conferences.sigcomm.org/sigcomm/2015/pdf/papers/p465.pdf

Thanks for the pointer.

Richard

---

From: Ingemar Johansson S <[email-masked]>
Date: Thu, 22 Mar 2018 10:59:51 +0000

Hi

With reference to #2. We have a similar issue in 5G where for instance LTE and NR is used in Dual Connectivity arrangements where :
1) The RTT over the NR leg is shorter than the RTT for the LTE leg. 
2) Data can be delayed an additional time on one leg because of retransmissions on the MAC and RLC layer.

We posted the following document a while ago (it was however not accepted). In short the proposed arrangement is that some mild reordering (say 10ms) is sorted out by the 5G protocol stack but beyond that the packets are forwarded out of order.
http://www.3gpp.org/ftp/tsg_ran/WG2_RL2/TSGR2_98/Docs/R2-1705318.zip

/Ingemar

> 
> Message: 3
> Date: Wed, 21 Mar 2018 03:23:31 -0700
> From: Yuchung Cheng <[email-masked]>
> To: Bob Briscoe <[email-masked]>
> Cc: [email-masked],  "[email-masked] Extensions"
> 	<[email-masked]>
> Subject: Re: [tcpm] Vicious or Virtuous circle? Adapting reordering
> 	window to reordering degree
> Message-ID:
> 	<CAK6E8=e3Eq49qPmSvDH3xBibKFAYQhHqkC_s3S=sOr2yFrfp_g@m
> ail.gmail.com>
> Content-Type: text/plain; charset="UTF-8"
> 
> On Tue, Mar 20, 2018 at 11:21 AM, Bob Briscoe <[email-masked]>
Snip snip....

> 
> We designed RACK based on studying the current traces we collected at
> Google. I see largely three kinds of reordering:
> 1) router parallelism. key symptom: the sub-MSS packet in the end of a
> window burst arrives first. the rest follows in order. e.g.
> P10,P1,P2,...P9. the reordering degree << min-RTT (except low BW)
> 2) wireless retransmission/burst at first/last hop: notably 802.11n block ACK
> may cause out-of-order delivery of TCP packets. Despite a reordering buffer
> is mandated in the spec, not all drivers implement that. symptom: random
> permutation of delivery but the degree in time is usually within the
> estimated RTT (e.g. SRTT) at the time of reordering.
> 3) the path is significantly shorten'd due to traffic engineering - the
> reordering degree is unbounded but happens infrequently (i.e. once many
> RTTs).
>