Analysis and Evaluation for TSN Queuing Mechanisms

2.1.1. Frame Preemption Overview

Frame preemption mechanism was introduced to mitigate negative effects of the guard band reserved by the TAS. As it requires modifications of both management (IEEE 802.1) and Ethernet MAC (IEEE 802.3) functions, two working groups jointly proposed required changes to both standards. Therefore, the frame preemption is described in two different standard documents: [IEEE802.1Qbu] and [IEEE802.3br].¶

[IEEE802.3br], also named Interspersing Express Traffic, differentiates two types of traffic: preemptable (also called mPacket) and express. The type of a frame is identified by examining the VLAN tag defined by IEEE 802.1Q. Frames arriving from the MAC client are serviced either by preemptable MAC (pMAC) or express MAC (eMAC). If both frames arrive at the same time, express traffic is serviced first as it has higher priority. In the case when express frame arrives while preemptable frame is already being transmitted on egress port, if certain conditions are met, it will interrupt current transmission . After express traffic has been serviced, the transmission of interrupted frame is resumed and different parts of the interrupted frame are re-assembled by a MAC Merge Sublayer (MMS) that is a part of the modified Ethernet MAC which supports frame preemption.¶

It is important to note that frame fragmentation works on link-by-link basis, i.e., each switch forwards preemptable frame only after it is fully re-assembled. This is clearly different from end-to-end packet fragmentation that is commonly used in IP networks. This ensures compatibility with the devices that do not support frame preemption mechanism.¶

2.1.2. Frame Preemption Analysis

As explained earlier, the main motivation behind the frame preemption mechanism is to reduce the length of guard band enforced by the TAS. Without frame preemption, reserved guard band must match the transmission time of the largest low-priority frame. In the case of 100 Mbps Ethernet, the worst-case time would be around 125us (transmission time of the largest Ethernet frame), which represents a huge bandwidth penalty.Frame preemption allows reducing the guard band down to approximately 12us which is tenfold improvement. It can also be combined with other queue technologies to minimize the interference delay from low priority packets.¶

2.2.1. CBS(CBS Overview)

CBS proposed by [IEEE802.1Qav] divides time-sensitive services which need to be transmitted preferentially into two classes: class A and class B, and sets a certain bandwidth for them. Through priority mapping, TSN flows with different priorities enter different queues for scheduling respectively. As described in Section 8.6.8.2 of [IEEE802.1Qav], the credits of each class increase according to the idle slope (as the guaranteed rate), and decrease according to the send slope (usually equal to idle slop minus port transmit rate), both of which are parameters of the CBS.¶

TSN flows are gently sent to the network by credit evaluation to deal with data burst and aggregation, CBS can limit burst traffic and prevent audio and video streams arriving at the same time from different terminals, which generates significant buffering congestion, resulting in packet loss.¶

2.2.2. CBS Analysis

CBS sets the pre-configuration of bandwidth limit for each traffic class. Typically set 75% of the maximum bandwidth for bandwidth intensive applications such as audio and video.¶

CBS does not rely on time synchronization, but still rely on frequency synchronization. So it can be applied in scenarios such as cross clock domains, non strict time synchronization, and asynchronous clocks.¶

The disadvantage of CBS is that the average latency will increase, although the combination of CBS and SRP (Stream Reservation Protocol) can limit the latency of each bridge to less than 250us. The paper[AVB-Latency] analyzes that in small-scale networks using FE (Fast Ethernet, 100Mbps) ports, CBS can guarantee a worst-case latency of less than 2 milliseconds for Class A and less than 50 milliseconds for Class B under a maximum of 7 hops. However, other papers[ClassA-Latency-Calc] shows the conclusion is not valid, it indicates that there is still a problem of delay degradation in CBS due to burstiness cascade. In general, the more hops, the worse the delay degradation . In large-scale networks, the number of network hops is usually large, such as 15 or more hops, which poses great challenges for the deployment of CBS independently. The upper bound of latency can not meet the requirements of many services which need low latency.¶

2.3.1. TAS Overview

In industrial IoT application scenarios, some time-sensitive streams will carry critical information. These streams require highly predictable delay and jitter in transmission. If the delay or jitter exceeds the threshold, it may cause serious consequences. At the same time, most of these streams are transmitted according to a certain time period, and streams with this characteristic are called Scheduled Traffic.¶

For the Scheduled Traffic, CBS transmission algorithm can not meet the requirements, because in CBS algorithm, if a low priority frame is already being transmitted, then that transmission will complete before a higher priority frame can access the transmission medium, so there could be a delay of up to a maximum-sized frame before a high priority transmission can start. If such delays occur at every hop, then the accumulated latency could be unacceptably large.¶

To address this issue, [IEEE802.1Qbv] proposes the TAS mechanism. As Scheduled Traffic is a periodic stream, it is possible to determine the time when each packet of streams arrives at each network device after Scheduled Traffic starts to transmit. As long as sufficient bandwidth is reserved for Scheduled Traffic on these devices in advance, it can ensure that other non-scheduled traffic will not interfere with the transmission of Scheduled Traffic.¶

TAS provides a scheduling mechanism of gate operations, which is based on high-precision clock synchronization. Each port of the TSN bridge has a gate control list (GCL) for opening or closing operations, and the 8 queues at these ports need to be associated with each of the 8 Transmission Gates respectively. Each entry in the GCL corresponds to a gate operation, and then packets are selected from the queue for transmission based on the gate control list. The gate control list contains two items: GateState and Timelnterval. GateState is used to set the state of Transmission Gate corresponding to queues, there are two states for each Transmission Gate: Open and Closed. "Open" means that packets in the associated queue can be transmitted according to the corresponding transmission algorithm, while "Closed" means that packets in the associated queue are not allowed to be transmitted. TimeInterval indicates the duration of the gate state. After TimeInterval ticks have elapsed since the completion of the previous gate operation in the GCL, control passes to the next gate operation.¶

Since transmission operation is an "atomic operation", in order to avoid the situation that the packet in corresponding queue can not be completely transmitted before the gate is closed, TAS defines an advanced check mechanism. If a packet cannot be fully transmitted within the remaining time of the corresponding gate operation state is open, this packet will not be transmitted until the next time when the gate is opened.¶

In order to ensure that the remaining non-scheduled traffic cannot affect the transmission of scheduled traffic, TAS uses a guard band (Guard Band) mechanism long enough to stop the transmission of non-scheduled traffic in advance of the protected time slot to be certain that the last non-scheduled transmission has completed before scheduled traffic transmission starts. In the worst case, the last non-scheduled transmission would start a maximum-sized frame transmission before the start of the scheduled traffic "window". In effect, a guard band is created before the time that the scheduled traffic transmission is due to start; transmission of non-scheduled traffic is not permitted between the start of the guard band and the start of the scheduled traffic window. The simplest approach for the guard band is to be as long as a maximum-sized frame transmission time.¶

2.3.2. TAS Analysis

The premise of TAS is that all terminals and network devices need to achieve nanosecond clock synchronization across the network (such as [IEEE1588], [IEEE802.1AS])to ensure that the GCL time of all outgoing ports is synchronized. Appropriate transmission "windows" can be arranged for the scheduled traffic at each outgoing port to achieve that the traffic can obtain extremely low transmission delay by accurate calculation. But when the network topology scale is large, that is, there may be a large number of nodes and links, it is usually difficult to achieve real-time synchronization, that limits the deployment of TAS; At the same time, large-scale networks carry a massive number of application flows, which will be a great challenge for TAS that relies on precise calculations and complex configurations.¶

On the other hand, the transmission window reserved for deterministic flows through GCL is usually exclusive. During the time period when the gate state of the queue associated with scheduled flows is open, even if the scheduled traffic does not arrive as expected, the transmission opportunities during this period will not be shared with other non-scheduled flows. Therefore, the bandwidth utilization in this scenario is insufficient.¶

2.4.1. CQF Overview

CQF follows the gate operations of the TAS mechanism: when the gate is open, the packets in the queue are allowed to be forwarded to the next node; when the gate is closed, incoming packets are buffered in the queue before they are allowed to transmit. CQF simplifies the design of TAS by installing fixed configurations on the GCL. Time in CQF networks is divided into cycles with equal value T, and there are two queues performing enqueue and dequeue operations in a cyclic manner under the control of RX GCL and TX GCL. When the packets enter the queue Q1 in cycle duration T(cycle c), the receiving gate of Q1 opens. Meanwhile, the output sending gate of Q2 opens, packets are transmitted to the next hop. When the next cycle(c+ 1) starts, the output sending gate of Q1 opens and sends the packets received in the previous cycle, the receiving gate of Q2 is open and starts to receive new packets. This cyclic queuing and forwarding mode can achieve transmission in a fixed duration that does not exceed 2T on per hop. CQF could provide the deterministic latency relies on two principles. First, the upstream and downstream nodes are perfectly synchronized, and the rotation of the upstream sending cycle and the downstream receiving cycle must be consistent. Second, a packet received at the cycle must be sent at the next cycle in a node. Thus, the predictable end-to-end latency only depends on the cycle size and path length, and regardless of topology. CQF is useful for applications that do not require very small latency and jitter, but which are still real-time and require bounded worst-case latency.¶

2.4.2. CQF Analysis

CQF can provide deterministic services with a maximum jitter of no more than 2T. The key issue is how to select the size of the cycle T and calculate the start time of the flow. The length of the queue is directly related to the size of cycle. If the cycle is too small, the queue is short too. Although the single hop queuing delay for traffic transmission is very small, there is not enough space to buffer more incoming flows, which can lead to a large number of flows that can not be scheduled; If the cycle is too large, it also means that queuing delay will become too large on per hop, which will result in a large end-to-end worst-case delay. Some traffic with requirements of low latency can not be scheduled, and larger queue lengths will also require more buffer resources. Due to limited underlying hardware resources, the difficulty and cost of hardware implementation are directly proportional to the buffer size.¶

It is necessary to carefully select the cycle size which needs to be large enough to accommodate all deterministic traffic, and in addition, the cycle includes a time duration called dead time (DT), which is the sum of delays 1, 2, 3 and 4 defined in Figure 1 of [RFC9320]. The value of DT ensures that the last packet of a cycle on the upstream node can be fully transmitted to the buffer of the same cycle on the downstream node. In the case of LAN, DT is relatively small compared to cycle T and is considered negligible, so only two buffer queues can run well. But in some deterministic networks, a single hop over a long distance can produce a large delay. Considering that the optical transmission speed in fiber is 200000km/s, the propagation delay of some long-distance links can be in the order of a few milliseconds, which is much larger than in LAN, and cannot be ignored. In order to cover the DT, more buffer queues need to be introduced.¶

On the other hand,the dead time (DT) reduces the available time for deterministic stream transmission within the cycle time, that make it impossible to deliver high-bandwidth services with extremely low jitter. Meanwhile, like TAS, classic CQF also rely on nanosecond clock synchronization across the entire network, where all network nodes align their cycle boundaries, and they cooperate with each other. This pattern limits the application of CQF in networks where precise time synchronization cannot be deployed.¶

In addition, a large amount of deterministic traffic demands will produce more fluctuations when dynamic services join or leave, which requires corresponding resource scheduling algorithms to allocate resources appropriately among multiple flows to avoid transmission conflicts. For example, in some complex aggregation situations, a large number of traffic with periodic characteristics may be gathered at a certain intermediate node. If the scheduling result is not appropriate, it will result in traffic congestion in one queue of the intermediate aggregation node, while the other part of the queue is idle, the traffic distribution is not balanced enough, which also exacerbates the probability of traffic conflicts. Currently CQF rely on overprovision to solve this problem, but this will result in a small scale of supported flows. Therefore, it is necessary to introduce optimized traffic planning design with path calculation and resource reservation, such as planning through a centralized controller, but it also puts higher requirements on the algorithm.¶

2.5.1. ECQF Overview

ECQF specifies procedures, protocols and managed objects for enhanced CQF, to avoid the requirement for system clock synchronization. ECQF specifies a transmission selection procedure that organizes frames in a traffic class output queue into logical bins that are output in strict rotation at a fixed frequency. It ensures that each bin will be emptied before the next bin is due for transmission. There are two ways of filling the bins: Time-based CQF stores received frames into bins based on the time of reception of the frame. Count-based CQF stores received frames into bins based on per-stream-per-output-queue byte counter state machines, and is recommended to use only on the boundary nodes of the frequency locking domain. Bin selection method can be configured based on an input-output port pair, or be configured for specific streams. It also provided multiple cycle model for different services, and the processing of flow aggregation/disaggregation based on count-based CQF.¶

ECQF is based on the following principles:¶

1. A Bridge output queue using ECQF is notionally divided into bins. The bins are enabled for output serially, at a fixed interval T_C, which same (or nearly the same) value is used for some number of Bridges along the path of a stream, said path constituting a ECQF segment of a network. At any given instant in time, a particular output bin can be available for accepting frames for later transmission, or enabled for transmitting frames to the associated medium, or neither, but never both.¶
2. Each stream utilizing a ECQF segment is allocated a certain number of bit times per transmission interval T_C. Steps are taken to ensure that no bin contains frames for any stream that will take, in total, longer than that stream’s allocated bit times to transmit. Resource reservation ensures that the total bit times allocated over all streams passing through a ECQF queue do not exceed T_C, even including possible interference from other queues on the port.¶
3. Frames assigned to the same bin at ingress to a ECQF Segment remain together in the same bin at each hop along the ECQF segment. Two methods are provided to accomplish this, time-based bin assignment and count-based bin assignment.¶

2.5.2. ECQF Analysis

ECQF does not rely on time synchronization. Time-based CQF need frequency lock and frequency synchronization. The number of CQF cycles in two Bridges that are frequency locked must be the same, over an arbitrarily long interval of time. Count-based CQF can be more relaxed. The cycle phase difference between two nodes is allowed. However, when the sum of clock jitter and phase difference exceeds N cycles (N is the number of selected bins), time-based CQF will not work.¶

In the ideal case, every stream would have a T_C value chosen so that exactly one frame of a stream is transmitted on each cycle T_C. Multiple values of T_C can be applied to a single output port. Streams are allocated to, and thus use up the bandwidth available to, each cycle separately. There are many ways to allocate buffer space to individual frames. Allocating bandwidth to a slower cycle times uses more buffer space, because frames dwell for a longer time. On the contrary, allocating bandwidth to a faster cycle time may get the optimal bounded latency, which may be somewhat oversubscribed. If the end-to-end latency requirements of the streams permit (but the case is not always like this), a stream can be assigned to a slower cycle. This will reduce the overprovision factor. Overprovision reduces the utilization of network resources.¶

Different CQF priority levels may operate simultaneously on one output port and have different maximum frame sizes, and some may enable preemption, different priority levels may have different amounts of time during one cycle that cannot be allocated to stream transmission. The burst may be limited at edge. For a new stream to be admitted, it must be true that the available transmission times over all of the CQF levels on all of the output ports through which the stream travels have not been exhausted.¶

2.6.1. ATS Overview

In order to solve the problem of zero congestion and packet loss in the transmission of aperiodic data, and to further optimize the bandwidth utilization of services without strict requirements for time synchronization, [IEEE802.1Qcr] defines an asynchronous traffic shaping device ATS.¶

ATS is designed based on Urgency-Based Scheduler ([UBS]). First,it identify the packets through the stream_handle (a sub-parameter of the stream identification function in [IEEE802.1CB]) and priority (the priority field in the VLAN tag) and match it into the corresponding stream filter, which specifies the stream gate and scheduler for the packets. The specified stream gate assigns internal priority, in this way, different degrees of delay guarantee can be provided in different nodes on the transmission path, it make the allocation of latency more flexible. The packets that has been assigned an internal priority enter the specified scheduler for shaping, which uses the interleaved algorithm based on the token bucket, and then assigned a eligible time, which is the expected transmission time of the packets. After the shaper, packets enter the corresponding shared queue (per incoming port plus traffic class) according to the internal priority and wait to be sent. The transmission selection algorithm is based on strict priority that transmits packets from the queues in the order from higher priority to lower priority sequentially. If the eligible time of the first packet in the shared queue is less than the current time, then the first packet can be sent directly and executes the transmission selection algorithm from the higher priority. Otherwise, turn to the next higher priority.¶

2.6.2. ATS Analysis

ATS adopts a principle called Rate-Controlled Service Disciplines (RCSD), which is a non work-conserving packet service discipline. It consists of two parts: the rate controller implements the rate control policy, and the scheduler implements packet scheduling based on some scheduling policy, such as static priority, first come first served, or earliest deadline. By separating the rate controller and the scheduler, RCSD effectively decouples the bandwidth of each stream from its delay bound, therefore, RCSD can support low latency and low bandwidth service.¶

The advantage of ATS is that when packets enter the queue, packets are assigned an eligible time, it allows urgent flows can be transmitted preferentially. ATS also has the concept of a scheduler group, where multiple ATS schedulers can belong to a single ATS scheduler group. The ATS scheduler does not rely on binding to hardware queues. From the delay analysis formula of ATS, it can be concluded that the allocation of internal priority and the assignation of scheduler directly determine the delay boundary of flows. In the stream gate component of each hop, different internal priority can be assigned to packets instead of external priority, that can more flexibly allocate the service level. Therefore, the ATS scheduler can perform flexible shaping for per flow or aggregated flows. ATS can be placed on each hop, then the network will not generate large burst accumulation, and the performance will be improved.¶

The ATS scheduler state machine operation is based on the ATS scheduler clocks, which is an implementation specific local system clock function. There is no need to require nodes in the network to achieve time synchronization.¶

A large number of flow aggregations will occur in a complex network topology, and it is necessary to consider flow aggregation strategies at intermediate nodes in the network. The end-to-end delay upper bound provided by ATS is generally inversely proportional to the service rate and may be larger.¶

Scaling deterministic networks require a large number of services to be carried, and the cost of interleaved regulators (IR) maintained in per hop is high. Meanwhile, it is necessary to pay attention to the problem of IR head-of-line blocking(HOL) in large-scale networks.¶