Subject: [tsvwg] On packet reordering in 5G

From: Ingemar Johansson S <[email-masked]>
Date: Sat, 3 Aug 2019 00:06:41 +0000

Hi

Long post and I probably missed something.. 

I was asked to clarify various aspects of packet reordering in 5G. I mentioned a little on this in  <https://lists.bufferbloat.net/pipermail/bloat/2019-March/008976.html> https://lists.bufferbloat.net/pipermail/bloat/2019-March/008976.html

I’ll try to keep the discussion around default bearers which are the typical bearers allocated for general mobile broadband (MBB) connectivity. Acknowledged mode (AM) transmission is used.

4G networks (and before this 3G) ensured in sequence delivery. The main reason behind this was that the TCP DupACK mechanism was sensitive to packet reordering. At that time it was likely a sensible thing to do as most of the traffic was mainly file transfers (and RACK was not discovered) 

Things have however changed over time.
1) Increased share of interactive services that require low latency. In-sequence delivery leads to HoL blocking issues and that often for no big use as transport protocols are better off to decide how to deal with out of order packets. This applies especially to QUIC, in which case many streams may be transmitted over one connection.

2) Higher link speeds means more memory is required in network nodes for the sole purpose of in-sequence delivery. For downlink traffic this means more memory in UE (User Equipment = USB modems, CPEs and other cellular devices). One can here argue that if in-sequence delivery is not enforced in on the RLC or PDCP layer, then it is likely needed to happen on the transport layer (e.g. TCP). But then again, for some transport protocols like QUIC where multiple streams are transmitted over the same connection, then it is likely better to get the packets delivered out of order.

First a short recap of the 3GPP user plane protocol stack (GTP stuff omitted), it goes like
| IP | PDCP | RLC | MAC | PHY | 

Where can packet reordering occur in 5G?

A.	On the MAC layer: Data from the RLC layer is put on transport blocks. If a transmission fails, a new transmission of the same data is repeated (but with a slightly different error control coding, incremental redundancy is used here to improve the spectrum efficiency. The max number of retransmits can be for instance 4, but this is configurable and vendor specific. Up to 16 HARQ (Hybrid AQR) processes can be run in NR (LTE has a limit of 8), this means that one chunk of data can be successfully transmitted before a previous chunk. This causes packet reordering. In LTE the reordering depth could be 8ms if one retransmission occurred, 16ms in case the same chunk of data is retransmitted twice and so on. In NR this reordering jitter is shorter, both as a result of a faster HARQ processing and shorter transmission intervals. As an example, with the highest numerology (0.125ms TTI=transmission interval) the jitter on the MAC layer is typically less than 1ms.
B.	On the RLC layer: The MAC layer transmission can fail if the max number of retransmission attempts is reached, this can occur if for instance the link adaptation (adjustment of the modulation and coding) is not optimal, in this case it is up to the RLC layer to retransmit the missing data. 
C.	Dual Connectivity: There can be cases where LTE is used for wide area coverage and NR is used for small cell coverage. One example is LTE running in the 2.4MHz band with 20MHz bandwidth and NR running on the 28GHz band with a 100MHz bandwidth. With Dual Connectivity, the data can be transmitted over both legs. Flow control is used to portion the data between the two legs. Depending on the efficiency of the  flow control and how the link bandwidth over the two legs vary, there will be packet reordering to a smaller or larger degree. I cannot give any example of typical reordering depths here as this reordering depends on the flow control algorithms in use.

In LTE it was up to the RLC layer to ensure in-sequence delivery up to higher layers. In 5G, this role is moved up to the PDCP layer, one reason is to make Dual Connectivity easier to implement. In addition, it is optional to implement in-sequence delivery. It is currently not possible for me to say if in-sequence delivery is implemented as it is vendor specific.

For the interested, this book gives more insight to the NR standard. 

 <https://www.elsevier.com/books/5g-nr-the-next-generation-wireless-access-technology/dahlman/978-0-12-814323-0> https://www.elsevier.com/books/5g-nr-the-next-generation-wireless-access-technology/dahlman/978-0-12-814323-0

OK, looks like shameless marketing but I can assure you that the authors (Ericsson employees) will pay me nothing for this ☹. Still the book is quite comprehensible and explains many of the design considerations.

Hope that this sheds at least some light

/Ingemar

================================

Ingemar Johansson  M.Sc. 

Master Researcher

Ericsson Research

Network Protocols & E2E Performance

Labratoriegränd 11

971 28, Luleå, Sweden

Phone +46-1071 43042

SMS/MMS +46-73 078 3289

[email-masked]

www.ericsson.com

            You’ve got to live with 

         What you can’t rise above
                Bruce Springsteen

=================================

---

From: Bob Briscoe <[email-masked]>
Date: Sat, 3 Aug 2019 14:20:40 +0100

Ingemar,

Thanks muchly for this comprehensive explanation of the state of the art.
I have posed a question for any folks from mobile operators watching 
this thread - inline.

On 03/08/2019 01:06, Ingemar Johansson S wrote:
> I was asked to clarify various aspects of packet reordering in 5G. I 
> mentioned a little on this in 
> https://lists.bufferbloat.net/pipermail/bloat/2019-March/008976.html
>
> I’ll try to keep the discussion around default bearers which are the 
> typical bearers allocated for general mobile broadband (MBB) 
> connectivity. Acknowledged mode (AM) transmission is used.
>
> 4G networks (and before this 3G) ensured in sequence delivery. The 
> main reason behind this was that the TCP DupACK mechanism was 
> sensitive to packet reordering. At that time it was likely a sensible 
> thing to do as most of the traffic was mainly file transfers (and RACK 
> was not discovered)
>
> 1) Increased share of interactive services that require low latency. 
> In-sequence delivery leads to HoL blocking issues and that often for 
> no big use as transport protocols are better off to decide how to deal 
> with out of order packets. This applies especially to QUIC, in which 
> case many streams may be transmitted over one connection.
>
> 2) Higher link speeds means more memory is required in network nodes 
> for the sole purpose of in-sequence delivery. For downlink traffic 
> this means more memory in UE (User Equipment = USB modems, CPEs and 
> other cellular devices). One can here argue that if in-sequence 
> delivery is not enforced in on the RLC or PDCP layer, then it is 
> likely needed to happen on the transport layer (e.g. TCP). But then 
> again, for some transport protocols like QUIC where multiple streams 
> are transmitted over the same connection, then it is likely better to 
> get the packets delivered out of order.
>
>  1. On the MAC layer: Data from the RLC layer is put on transport
>     blocks. If a transmission fails, a new transmission of the same
>     data is repeated (but with a slightly different error control
>     coding, incremental redundancy is used here to improve the
>     spectrum efficiency. The max number of retransmits can be for
>     instance 4, but this is configurable and vendor specific. Up to 16
>     HARQ (Hybrid AQR) processes can be run in NR (LTE has a limit of
>     8), this means that one chunk of data can be successfully
>     transmitted before a previous chunk. This causes packet
>     reordering. In LTE the reordering depth could be 8ms if one
>     retransmission occurred, 16ms in case the same chunk of data is
>     retransmitted twice and so on. In NR this reordering jitter is
>     shorter, both as a result of a faster HARQ processing and shorter
>     transmission intervals. As an example, with the highest numerology
>     (0.125ms TTI=transmission interval) the jitter on the MAC layer is
>     typically less than 1ms.
>  2. On the RLC layer: The MAC layer transmission can fail if the max
>     number of retransmission attempts is reached, this can occur if
>     for instance the link adaptation (adjustment of the modulation and
>     coding) is not optimal, in this case it is up to the RLC layer to
>     retransmit the missing data.
>  3. Dual Connectivity: There can be cases where LTE is used for wide
>     area coverage and NR is used for small cell coverage. One example
>     is LTE running in the 2.4MHz band with 20MHz bandwidth and NR
>     running on the 28GHz band with a 100MHz bandwidth. With Dual
>     Connectivity, the data can be transmitted over both legs. Flow
>     control is used to portion the data between the two legs.
>     Depending on the efficiency of the  flow control and how the link
>     bandwidth over the two legs vary, there will be packet reordering
>     to a smaller or larger degree. I cannot give any example of
>     typical reordering depths here as this reordering depends on the
>     flow control algorithms in use.
>
> In LTE it was up to the RLC layer to ensure in-sequence delivery up to 
> higher layers. In 5G, this role is moved up to the PDCP layer, one 
> reason is to make Dual Connectivity easier to implement.
>
And this makes more sense once the RLC frames map 1:1 to IP packets in 5G.

> In addition, it is optional to implement in-sequence delivery. It is 
> currently not possible for me to say if in-sequence delivery is 
> implemented as it is vendor specific.
>
Assuming equipment vendors implement in-sequence delivery, the real 
question is whether mobile operators will still want it enabled on the 
default (MBB) bearer. Does anyone (e.g. from an operator) have a feel 
for an answer?

I assume that operators will keep PDCP in-sequence delivery enabled 
until the prevalence of modern re-ordering tolerant transports combined 
with the benefits to them outweigh the prevalence and detriments to 
conventional (3DupACK) transports.

This messy compromise is why we took the opportunity to draw a line in 
the sand with L4S, by requiring sender algorithms that have to be 
modified to support L4S to also have modern reordering tolerance from day-1.

I know there are objections on editorial, architectural and procedural 
grounds, so I shall be working through that minefield next week, because 
I've seen a lot of support on technical and pragmatic grounds - the two 
types are hard to weigh against each other.

> https://www.elsevier.com/books/5g-nr-the-next-generation-wireless-access-technology/dahlman/978-0-12-814323-0
>
> OK, looks like shameless marketing but I can assure you that the 
> authors (Ericsson employees) will pay me nothing for this ☹. Still the 
> book is quite comprehensible and explains many of the design 
> considerations.
>
Indeed it does. Thank you again.

Bob

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/