Subject: Re: [Detnet] new draft on segment routing approach to TSN

From: Haoyu Song <[email-masked]>
Date: Fri, 5 Mar 2021 20:45:45 +0000

Hi Yaakov,

Just got a chance to read your draft. I agree with the comments of the others that this is a very interesting work. I'll just add a few points.

  1.  The use of clock time as deadline requires network synchronization, and accurate measurement of per-link propagation time, which can somehow limit the application scope of this work. Alternatively, one can simply budget a device latency which require a router/switch to obey. In case the overall budget is evenly divided by the hops, a single parameter is enough. Of course, if one wants to customize the budget on each hop (which might be necessary considering the different capability/capacity of each hop), a stack is still needed.
  2.  Mechanism should be provisioned to track where the timing requirement is violated and by how much (e.g., using timestamp or flag). This would be very useful for troubleshooting.
  3.  Recently programmable scheduler research has proposed several primitives such as PIPO and PIEO and provided feasible hardware implementations. The scheme proposed in this draft can easily fit into these primitives.

Best regards,
Haoyu
From: spring <[email-masked]> On Behalf Of Yaakov Stein
Sent: Tuesday, February 23, 2021 5:14 AM
To: [email-masked]; [email-masked]; [email-masked]
Subject: [spring] new draft on segment routing approach to TSN

All,

I would like to call your attention to a new ID https://www.ietf.org/archive/id/draft-stein-srtsn-00.txt<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ietf.org%2Farchive%2Fid%2Fdraft-stein-srtsn-00.txt&data=04%7C01%7Chaoyu.song%40futurewei.com%7C603ba064d1ef4220fa0508d8d7fd0e0b%7C0fee8ff2a3b240189c753a1d5591fedc%7C1%7C1%7C637496829160448349%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=Fq7J2ogbHh%2B26TZ2dRc82eUhJkwAOSWdnvR1G68r5p8%3D&reserved=0>
which describes using a stack-based approach (similar to segment routing) to time sensitive networking.
It furthermore proposes combining segment routing with this approach to TSN
resulting in a unified approach to forwarding and scheduling.

The draft is information at this point, since it discusses the concepts and does not yet pin down the precise formats.

Apologies for simultaneously sending to 3 lists,
but I am not sure which WG is the most appropriate for discussions of this topic.

  *   DetNet is most relevant since the whole point is to control end-to-end latency of a time-sensitive flow.
  *   Spring is also directly relevant due to the use of a stack in the header and the combined approach just mentioned.
  *   PCE is relevant to the case of a central server jointly computing an optimal path and local deadline stack.
I'll let the chairs decide where discussions should be held.

Y(J)S

---

From: Yaakov Stein <[email-masked]>
Date: Sun, 7 Mar 2021 04:35:30 +0000

Haoyu,

I'll address your points:

> The use of clock time as deadline requires network synchronization
That is a basic assumption of TSN networks (IMO the defining assumption).
However, the sync needn't be highly accurate - it depends on the tightness of the delay budgets.

> accurate measurement of per-link propagation time
Yes, I am assuming that once an for all someone does a TDR or at least OWAMP/TWAMP/Y.1731-1wD delay measurement of the links, and these are stored in some database.
Once again, the required accuracy depends on the delay budgets.

> which can somehow limit the application scope of this work
If the delay budget only above the physically minimal delay by say less than 100 microseconds,
I agree that the previous two issues MUST be carried out. But in such cases there is no alternative.
If the delay budget is much higher than that, then one could use an RSVP-like mechanism,
sending a packet (or several packets) from source to destination collecting a stack of timestamps,
and then using that stack for the following packets.

> Mechanism should be provisioned to track where the timing requirement is violated and by how much
I'll leave the OAM for later. However there are already many high accuracy performance measurement techniques and protocols for this.

> Recently programmable scheduler research has proposed several primitives
Yes, I tried to stress that this ID is not limited to EDF (although sometimes that is a good strategy).
One can even reproduce Qbv behavior using a stack of deadlines (although why would one wish to do so?).

> such as PIPO and PIEO
I've heard of PIFO (Push In First out) but not PIPO. Is this a typo or something new?
I agree that there are mechanisms that are optimized for hardware, but I have come up with a very nice hardware implementation for PEDF
and prefer to find hardware implementations for optimal schedulers, rather than to determine schedulers based on optimal hardware.

Y(J)S

From: Haoyu Song <[email-masked]>
Sent: 05/03/2021 22:46
To: Yaakov Stein <[email-masked]>; [email-masked]; [email-masked]; [email-masked]
Subject: RE: new draft on segment routing approach to TSN

CAUTION: External sender. Do not click links or open attachments unless you know the content is safe.
Hi Yaakov,

Best regards,
Haoyu
From: spring <[email-masked]<mailto:[email-masked]>> On Behalf Of Yaakov Stein
Sent: Tuesday, February 23, 2021 5:14 AM
To: [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: [spring] new draft on segment routing approach to TSN

All,

I would like to call your attention to a new ID https://www.ietf.org/archive/id/draft-stein-srtsn-00.txt<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ietf.org%2Farchive%2Fid%2Fdraft-stein-srtsn-00.txt&data=04%7C01%7Cyaakov_s%40rad.com%7C29e393d86da94a8c90b908d8e017a95d%7Cf9047108cc2c4e4897a343fad1b3bf9d%7C1%7C0%7C637505739530106136%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=Is%2F58aDPKY6V%2FBEVAF5getSEUvLgyDjInxJNT%2BmpnWM%3D&reserved=0>
which describes using a stack-based approach (similar to segment routing) to time sensitive networking.
It furthermore proposes combining segment routing with this approach to TSN
resulting in a unified approach to forwarding and scheduling.

Y(J)S

---

From: Haoyu Song <[email-masked]>
Date: Mon, 8 Mar 2021 02:41:17 +0000

Hi Yaakov,

Some feedback inline.

Best regards,
Haoyu

From: Yaakov Stein <[email-masked]>
Sent: Saturday, March 6, 2021 8:36 PM
To: Haoyu Song <[email-masked]>; [email-masked]; [email-masked]; [email-masked]
Subject: RE: new draft on segment routing approach to TSN

Haoyu,

>> Yes, I understand it. What I mean is that the method mentioned in the draft seems to be also applicable to other types of networks. For example, we can envision some time-critical traffic in DCN or WAN. If a protocol would be developed, can it also serve the other networks? If so, it would be better.

>> For this I mean something recorded in the same packet with the deadlines. If it misses the deadline, the receiver may need to know where it's violated. Other independent methods are possible, but it's better to consider if it can be integrated in the current proposal.

>> Sorry that's a typo. I mean PIFO (although we do have a paper under review using the name PIPO). Yes I agree those are just abstract primitives. The actual implementation, if customized to a particular algorithm, would be simpler.

Y(J)S

From: Haoyu Song <[email-masked]<mailto:[email-masked]>>
Sent: 05/03/2021 22:46
To: Yaakov Stein <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: RE: new draft on segment routing approach to TSN

Best regards,
Haoyu

All,

I would like to call your attention to a new ID https://www.ietf.org/archive/id/draft-stein-srtsn-00.txt<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ietf.org%2Farchive%2Fid%2Fdraft-stein-srtsn-00.txt&data=04%7C01%7Chaoyu.song%40futurewei.com%7C6aeaab604ad842c5ca0f08d8e122721c%7C0fee8ff2a3b240189c753a1d5591fedc%7C1%7C0%7C637506885364131187%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=ZHZdFoS63OmpW1tEcp9BBksGzRGfs8IF9NpJnS%2FlXa8%3D&reserved=0>
which describes using a stack-based approach (similar to segment routing) to time sensitive networking.
It furthermore proposes combining segment routing with this approach to TSN
resulting in a unified approach to forwarding and scheduling.

Y(J)S

---

From: Yaakov Stein <[email-masked]>
Date: Mon, 8 Mar 2021 05:37:05 +0000

Haoyu

I think we are in agreement.

I do not see the need for explicitly handling the case of a packet missing a LOCAL deadline,
since the following switches will already handle this case optimally (and may still meet the overall budget).

Counting packets that miss their delay budget is indeed important,
and a counter could be configured in the egress router for this.
We'll need to define this when we get to the protocol specification.

It would be advantageous to put a threshold on the failure rate
and feed this back to the path/stack optimizer.

Y(J)S

Sent: 08/03/2021 04:41

Hi Yaakov,

Some feedback inline.

Best regards,
Haoyu

From: Yaakov Stein <[email-masked]<mailto:[email-masked]>>
Sent: Saturday, March 6, 2021 8:36 PM
To: Haoyu Song <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: RE: new draft on segment routing approach to TSN

Haoyu,

Y(J)S

Best regards,
Haoyu

All,

I would like to call your attention to a new ID https://www.ietf.org/archive/id/draft-stein-srtsn-00.txt<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ietf.org%2Farchive%2Fid%2Fdraft-stein-srtsn-00.txt&data=04%7C01%7Cyaakov_s%40rad.com%7Ccf886b3eaea04f141b4308d8e1dba873%7Cf9047108cc2c4e4897a343fad1b3bf9d%7C1%7C0%7C637507680849467802%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=6TSHluQTQh2IyoehtQcGCjqxtT2MlJtQycz%2FCHVno74%3D&reserved=0>
which describes using a stack-based approach (similar to segment routing) to time sensitive networking.
It furthermore proposes combining segment routing with this approach to TSN
resulting in a unified approach to forwarding and scheduling.

Y(J)S

---

From: Balázs Varga A <[email-masked]>
Date: Mon, 8 Mar 2021 06:35:24 +0000

Hi Yaakov,
Any view on these comments?
Thanks
Bala'zs

From: detnet <[email-masked]> On Behalf Of Balázs Varga A
Sent: Friday, February 26, 2021 3:03 PM
To: Yaakov Stein <[email-masked]>; [email-masked]; [email-masked]; [email-masked]
Subject: Re: [Detnet] new draft on segment routing approach to TSN

Hi Yaakov,

many thanks for this well-written and information-rich draft.
I have enjoyed to read it (and also the discussions on the lists). :--)))

Conceptual (clarification) questions:
1, What is the node behavior if deadtime expires? Drop? That requires a quite good design/allocation of the latency budget. Otherwise it may result in unnecessary loss of a packet being relative late at a hop, however that might be compensated by the remaining hops. (E.g., see your Figure 2, getting 35 microsecond at R1 instead of the designed 25, but R2,R3,R4 adds only 10-10-10 extra microseconds). You may consider to add a "late flag" to the packet instead of a drop ...

2, Do I understand it correctly that the described solution is practically a new per-hop-behavior? Does it have no tools to "influence" the arrival time at the next hops? In end-to-end latency design the DetNet/TSN functions are often used to "adapt" the arrival of packets belonging to a DetNet flow/TSN stream at given network points. If "latency design" requires such a "time shift" then the described method has to be combined with other DetNet/TSN tools.

3, Design of a guaranteed upper bound usually requires deterministic arrival at each node along the path. In this solution the flow becomes less-an-less deterministic as it reaches the destination. I mean e.g., as per Figure 2, arrival time range at R1 is 2-2 usec, at R2 is 26-51 usec, at R3 is 70-120 usec and at R4 is 92-167 usec. So, the flow is getting less-and-less deterministic at each hop. That can make deterministic design of other flows passing R4 more difficult or even impossible in some scenarios. Have I missed something here?

4, Wire-speed timestamping and calculation: I would be interested in your view regarding how realistic are (1) the wire speed timestamping capability and (2) adding multiple calculated deadline to packets. Both are required for this solution. DetNet flows can have high bandwidth, not like 1588 packets.

Specific comments (Section 2-3):
5, Time gated Queuing [802.1Qbv] allows multiple gates to be open simultaneously. Furthermore preemption [802.1Qbu] can be combined with Qbv. So I cannot fully agree with your evaluation of time-aware scheduling. (Section 2, "However, time-aware scheduling suffers from two major disadvantages.") Of course with a bad design one may shoot in his/her own leg, but in well-designed scenarios efficiency loss can be eliminated without expensive computation.

6, Deadtime calculation is also complex if impact of other DetNet flows must be considered. The pre-calculation of individual "local" deadlines may need to be re-calculated when flows added to the network and they are using some common links. So I cannot agree with your statement. (Section 3, "... Since the ingress router inserts the deadline stack into the packet headers, no other router needs to be aware of the requirements of the time sensitive flows.  Hence admitting a new flow only requires updating the information base of the ingress router."). Adding a flow may result in updating many ingress routers' configuration.

Some topics for further discussions/considerations:
7, Impact of some DetNet functions (e.g., PREOF [RFC8655]) needs further discussions. The usage of replication/elimination impacts the deadline calculation. Disjoint paths used by member flows requires different label stack and deadlines. Furthermore, these path specific labels+deadlines must be added by the replication node (PRF). So at least, a combination of PRF with B-SID and computing/adding deadlines (ingress SRTSN function) seems to be necessary in your solution.

8, The ever-green topic ( and not the green-wave :--))) ): label stack size. You need multiple labels per hop and the label stack contains them for each hop. Could result in a quite big label stack to be pushed at the ingress (nx10s of labels).

I am happy to have further discussions on this interesting idea

Thanks & Cheers
Bala'zs

From: detnet <[email-masked]<mailto:[email-masked]>> On Behalf Of Yaakov Stein
Sent: Tuesday, February 23, 2021 2:14 PM
To: [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: [Detnet] new draft on segment routing approach to TSN

All,

I would like to call your attention to a new ID https://www.ietf.org/archive/id/draft-stein-srtsn-00.txt
which describes using a stack-based approach (similar to segment routing) to time sensitive networking.
It furthermore proposes combining segment routing with this approach to TSN
resulting in a unified approach to forwarding and scheduling.

Y(J)S

---

From: Yaakov Stein <[email-masked]>
Date: Mon, 8 Mar 2021 07:26:45 +0000

Bala'zs,

Sorry that I never got back to you on this.
It is much easier to shoot back replies to shorter emails :)

> 1, What is the node behavior if deadtime expires?
Nothing special is done if a local deadline is exceeded. There is still hope that the time can be made up.
If too many packets fail to conform to their overall budget,
then maybe something has changed and reoptimization (or realizing that the application budget is not feasible) is required.

> 2, Do I understand it correctly that the described solution is practically a new per-hop-behavior?
Well, new to here but hardly "new" (similar things were done in ATM...)
The new per-router behavior requires a new data structure in the router instead of a FIFO queue,
but there is no absolute requirement for other tools.
As I just hinted and have mentioned in a previous email, later on we may need some OAM to see if configuration needs to be updated.
I have already experimented with periodic RSVP-like synthetic packets that collect timestamps along the path.

> 3, Design of a guaranteed upper bound usually requires deterministic arrival at each node along the path.
Here is where we differ (or at least put different emphasis on the word "usually".

Determinism is not the main goal of this draft. Meeting challenging delay budgets is.
Sometimes absolute determinacy is required (e.g., in some forms of "teleprotection" in some electricity distribution networks)
in which case either TDM/Qbv/calendaring can be used, or this method coupled with a differential delay compensation buffer.

I agree that PDV increases from hop to hop to an extent, but when the delay budget is really challenging you simply can't allow that many hops.
In mixed 5G xHaul cases in which I have been involved there are no more than 4 hops (usually fewer).

If you wish something more deterministic then look at the Andrews-Zhang strategy.
They gate at the ingress router and EDF at all the following ones.
They show that (statistically) they ride "green waves" and thus the PDV doesn't increase significantly
(see their graphs). This comes at the expense of a long cycle time for the initial gate, which means the budget can't be too tight.

> 4, Wire-speed timestamping and calculation
I come from the PTP world where hardware timestamping with submicro resolution is common.
Since the timestamp is on the rising edge of the first bit, i.e., before the classifier,
it is carried out and placed in the packet metadata for every packet whether or not it is PTP.
I always shed tears over throwing out this information.
The RSVP-like method just mentioned simply copies it into the packet.
The SRTSN packets use this information in the scheduler.
I have consulted my ASIC people about the use of priority heaps and insert-queues
and they ensure me that they can put something like that into our 100G chip.

> 5, Time gated Queuing [802.1Qbv]
I didn't say that in principle you can't build a Qbv system that does great things.
It is flexible enough to do anything you want.
The problem is the scalability.
I have played around with optimizing it and haven't found a good "on line" algorithm,
that is, a way of adding a new flow to an existing optimized design.
And even in the "off line" case (clean slate with all flows known) slight changes and even bad clocks in the user equipment ruins things.

But the main idea of this draft is that even given a good on-line algorithm for Qbv
you would need to distribute the changes in the all the schedules to all the routers.
What happens in the meantime? What happens when one router is already updated but another isn't?
With the stack method only a single ingress router needs to be updated and consistency is ensured!

> 6, Deadtime calculation is also complex if impact of other DetNet flows must be considered.
Of course! The offset vector needs to be optimized.
I have several such methods and am running simulations of various strategies.
As stated in the ID the specific calculation was for illustration purposes -
the stack mechanism is much more generic.

> 7, Impact of some DetNet functions
I have not yet gotten to the point where I have considered this.

>  8, The ever-green topic ( and not the green-wave :--))) ): label stack size.
Yes, I didn't want to bring this up yet, but have been forced to.
I have to admit that I am not a fan of SRv6 for just this reason.
But, the SRTSN stack can be really small!
SR stack entries need only be ceil ( log2 (number of routers) ) in size (requires a little local information base in the router)
assuming once the stack is popped you use the original DA. If you are not willing to use this additional information
you still only need sufficient suffixes, which is still small.
The deadline entries can be miniscule since the wraparound constraint is derived from the maximum flight time of a packet in the network.
In my designs I have gotten away with 16-32 bits per entry, which means 4-8 hop stacks only require the room of a single IPv6 address.

Hope this answers your questions and kicks off a fruitful discussion.

Y(J)S

From: Balázs Varga A <[email-masked]>
Sent: 08/03/2021 08:35
To: Yaakov Stein <[email-masked]>; [email-masked]; [email-masked]; [email-masked]
Subject: RE: new draft on segment routing approach to TSN

From: detnet <[email-masked]<mailto:[email-masked]>> On Behalf Of Balázs Varga A
Sent: Friday, February 26, 2021 3:03 PM
To: Yaakov Stein <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: Re: [Detnet] new draft on segment routing approach to TSN

Hi Yaakov,

Thanks & Cheers
Bala'zs

All,

I would like to call your attention to a new ID https://www.ietf.org/archive/id/draft-stein-srtsn-00.txt<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ietf.org%2Farchive%2Fid%2Fdraft-stein-srtsn-00.txt&data=04%7C01%7Cyaakov_s%40rad.com%7C270d392bafcf4742c92d08d8e1fc5c56%7Cf9047108cc2c4e4897a343fad1b3bf9d%7C1%7C0%7C637507821310056080%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=o332wbi4u5fb%2BKtUW1CsB3jz9OEd32h%2BazNaEDnDHJY%3D&reserved=0>
which describes using a stack-based approach (similar to segment routing) to time sensitive networking.
It furthermore proposes combining segment routing with this approach to TSN
resulting in a unified approach to forwarding and scheduling.

Y(J)S

---

From: Balázs Varga A <[email-masked]>
Date: Mon, 8 Mar 2021 09:58:56 +0000

Hi Yaakov,

Thanks for the clarifications, that helps a lot to position correctly your draft.
As You have wrote: "Determinism is not the main goal of this draft" what
was not clear from the text.

I may be somewhat mis-leaded by the name You have used for the concept.
TSN is widely used in IEEE as "Time-Sensitive Networking" and its main target
is deterministic communication.

You may consider to use a different abbreviation to avoid that confusion. E.g.,
"srtsc" (segment routing time sensitive communication) or anything else You like.

Many thanks
Bala'zs

From: Yaakov Stein <[email-masked]>
Sent: Monday, March 8, 2021 8:27 AM
To: Balázs Varga A <[email-masked]>; [email-masked]; [email-masked]; [email-masked]
Subject: RE: new draft on segment routing approach to TSN

Bala'zs,

Y(J)S

From: Balázs Varga A <[email-masked]<mailto:[email-masked]>>
Sent: 08/03/2021 08:35
To: Yaakov Stein <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: RE: new draft on segment routing approach to TSN

Hi Yaakov,

Thanks & Cheers
Bala'zs

All,

I would like to call your attention to a new ID https://www.ietf.org/archive/id/draft-stein-srtsn-00.txt<https://protect2.fireeye.com/v1/url?k=c3902149-9c0b184c-c39061d2-86d2114eab2f-ba16ddb9b86334c8&q=1&e=25a393fd-8aca-4a3b-914c-6ce612f6740b&u=https%3A%2F%2Feur01.safelinks.protection.outlook.com%2F%3Furl%3Dhttps%253A%252F%252Fwww.ietf.org%252Farchive%252Fid%252Fdraft-stein-srtsn-00.txt%26data%3D04%257C01%257Cyaakov_s%2540rad.com%257C270d392bafcf4742c92d08d8e1fc5c56%257Cf9047108cc2c4e4897a343fad1b3bf9d%257C1%257C0%257C637507821310056080%257CUnknown%257CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%253D%257C1000%26sdata%3Do332wbi4u5fb%252BKtUW1CsB3jz9OEd32h%252BazNaEDnDHJY%253D%26reserved%3D0>
which describes using a stack-based approach (similar to segment routing) to time sensitive networking.
It furthermore proposes combining segment routing with this approach to TSN
resulting in a unified approach to forwarding and scheduling.

Y(J)S

---

From: Yaakov Stein <[email-masked]>
Date: Mon, 8 Mar 2021 11:45:20 +0000

Bala'zs,

The IEEE definition of TSN mentions determinism, but defines it not as constant delay as in a TDM service, but rather as
to provide deterministic services through IEEE 802 networks,
i.e., guaranteed packet transport with bounded latency, low packet delay variation, and low packet loss.
So "determinism" in this context means bounded latency, low PDV and low PLR.

DetNet is aligned with this definition. From the charter:
deterministic data paths that operate over Layer 2 bridged and Layer 3 routed segments, where such paths can provide bounds on latency, loss, and packet delay variation (jitter), and high reliability.
So it is the path that is deterministic, but the packet parameters need to be bounded.

In Carrier grade Ethernet "determinism" is usually taken to mean that a flow is pinned to a path,
that is, we know precisely which network elements a packet will traverse,
although delay, PLR, etc. are parameters to be set and/or measured.

When I say that a packet or flow is "time sensitive" I merely mean that there is a time budget to be attained.

However, SRTSN by any other name smells as sweet to me.
Stack-based Time-sensitive Enhancements for Immediate-delivery Networking (STEIN) could also work :) .

Y(J)S

Sent: 08/03/2021 11:59

Hi Yaakov,

Many thanks
Bala'zs

From: Yaakov Stein <[email-masked]<mailto:[email-masked]>>
Sent: Monday, March 8, 2021 8:27 AM
To: Balázs Varga A <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: RE: new draft on segment routing approach to TSN

Bala'zs,

Y(J)S

Hi Yaakov,

Thanks & Cheers
Bala'zs

All,

I would like to call your attention to a new ID https://www.ietf.org/archive/id/draft-stein-srtsn-00.txt<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fprotect2.fireeye.com%2Fv1%2Furl%3Fk%3Dc3902149-9c0b184c-c39061d2-86d2114eab2f-ba16ddb9b86334c8%26q%3D1%26e%3D25a393fd-8aca-4a3b-914c-6ce612f6740b%26u%3Dhttps%253A%252F%252Feur01.safelinks.protection.outlook.com%252F%253Furl%253Dhttps%25253A%25252F%25252Fwww.ietf.org%25252Farchive%25252Fid%25252Fdraft-stein-srtsn-00.txt%2526data%253D04%25257C01%25257Cyaakov_s%252540rad.com%25257C270d392bafcf4742c92d08d8e1fc5c56%25257Cf9047108cc2c4e4897a343fad1b3bf9d%25257C1%25257C0%25257C637507821310056080%25257CUnknown%25257CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%25253D%25257C1000%2526sdata%253Do332wbi4u5fb%25252BKtUW1CsB3jz9OEd32h%25252BazNaEDnDHJY%25253D%2526reserved%253D0&data=04%7C01%7Cyaakov_s%40rad.com%7C1b7b95bc875e48d5a3fe08d8e218cc62%7Cf9047108cc2c4e4897a343fad1b3bf9d%7C1%7C0%7C637507943449085449%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=Q%2BfWafpslb%2BsCSQrmBzcA2fhopPw3GAj9me0v7xqdk0%3D&reserved=0>
which describes using a stack-based approach (similar to segment routing) to time sensitive networking.
It furthermore proposes combining segment routing with this approach to TSN
resulting in a unified approach to forwarding and scheduling.

Y(J)S

---

From: "Liubingyang (Bryan)" <[email-masked]>
Date: Mon, 8 Mar 2021 12:54:34 +0000

Hi Yaakov,

Great work. One suggestion.
If we divide the end-to-end delay budget to per-hop budgets, and carry them in SR header, we can mimic using the end-to-end absolute deadline. One thing we need to do is, when the actual time used in a hop is less than the budget of this hop, you can add the remaining budget to the budget of next hop. In this case, you can save a lot of bits used for time, and you don't have to rely on accurate time sync.

Bryan (Bingyang Liu)

From: detnet [mailto:[email-masked]] On Behalf Of Yaakov Stein
Sent: Monday, March 8, 2021 1:37 PM
To: Haoyu Song <[email-masked]>; [email-masked]; [email-masked]; [email-masked]
Subject: Re: [Detnet] new draft on segment routing approach to TSN

Haoyu

Y(J)S

Hi Yaakov,

Some feedback inline.

Best regards,
Haoyu

Haoyu,

Y(J)S

Hi Yaakov,

1)      The use of clock time as deadline requires network synchronization, and accurate measurement of per-link propagation time, which can somehow limit the application scope of this work. Alternatively, one can simply budget a device latency which require a router/switch to obey. In case the overall budget is evenly divided by the hops, a single parameter is enough. Of course, if one wants to customize the budget on each hop (which might be necessary considering the different capability/capacity of each hop), a stack is still needed.
2)      Mechanism should be provisioned to track where the timing requirement is violated and by how much (e.g., using timestamp or flag). This would be very useful for troubleshooting.
3)      Recently programmable scheduler research has proposed several primitives such as PIPO and PIEO and provided feasible hardware implementations. The scheme proposed in this draft can easily fit into these primitives.

Best regards,
Haoyu

All,

Y(J)S

---

From: Yaakov Stein <[email-masked]>
Date: Mon, 8 Mar 2021 13:37:55 +0000

Bryan

The local deadlines are specified in absolute time - not deltas.
So there is no need to add the time saved - it is already there!

As I previously replied, the accuracy of the time sync depends on the tightness of the budget.
It has nothing to do with the encoding.

Y(J)S

From: Liubingyang (Bryan) <[email-masked]>
Sent: 08/03/2021 14:55
To: Yaakov Stein <[email-masked]>; Haoyu Song <[email-masked]>; [email-masked]; [email-masked]; [email-masked]
Subject: RE: new draft on segment routing approach to TSN

Hi Yaakov,

Bryan (Bingyang Liu)

Haoyu

Y(J)S

Hi Yaakov,

Some feedback inline.

Best regards,
Haoyu

Haoyu,

Y(J)S

Best regards,
Haoyu

All,

I would like to call your attention to a new ID https://www.ietf.org/archive/id/draft-stein-srtsn-00.txt<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ietf.org%2Farchive%2Fid%2Fdraft-stein-srtsn-00.txt&data=04%7C01%7Cyaakov_s%40rad.com%7C71c93afad8614e5d212e08d8e23156c8%7Cf9047108cc2c4e4897a343fad1b3bf9d%7C1%7C0%7C637508048843821932%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=J5d%2Bz%2BmTnTm3C7%2FfD0YSo%2FXMk%2F67YxHlYVRfvv%2Fthcg%3D&reserved=0>
which describes using a stack-based approach (similar to segment routing) to time sensitive networking.
It furthermore proposes combining segment routing with this approach to TSN
resulting in a unified approach to forwarding and scheduling.

Y(J)S

---

From: "Liubingyang (Bryan)" <[email-masked]>
Date: Mon, 8 Mar 2021 14:30:37 +0000

Some advantages.
If you use budget instead of deadline, you don't need accurate time sync.
If you use budget, the time budget encoded in packet require fewer bits than using absolute time.

The tightness is related to whether and how many flows can be admitted. It is related to network resource utilization and revenue.

If e2e deadline is 1ms, then time sync with 1ms accuracy will not work well. Using budget helps resolve this problem.

Just my two cents.

Bryan

From:Yaakov Stein <[email-masked]>
To:Liubingyang (Bryan) <[email-masked]>;Haoyu Song <[email-masked]>;detnet <[email-masked]>;spring <[email-masked]>;pce <[email-masked]>
Date:2021-03-08 21:38:20
Subject:RE: new draft on segment routing approach to TSN

Bryan

The local deadlines are specified in absolute time – not deltas.
So there is no need to add the time saved – it is already there!

Y(J)S

Hi Yaakov,

Great work. One suggestion.
If we divide the end-to-end delay budget to per-hop budgets, and carry them in SR header, we can mimic using the end-to-end absolute deadline. One thing we need to do is, when the actual time used in a hop is less than the budget of this hop, you can add the remaining budget to the budget of next hop. In this case, you can save a lot of bits used for time, and you don’t have to rely on accurate time sync.

Bryan (Bingyang Liu)

Haoyu

Counting packets that miss their delay budget is indeed important,
and a counter could be configured in the egress router for this.
We’ll need to define this when we get to the protocol specification.

Y(J)S

Hi Yaakov,

Some feedback inline.

Best regards,
Haoyu

Haoyu,

That is a basic assumption of TSN networks (IMO the defining assumption).
However, the sync needn’t be highly accurate – it depends on the tightness of the delay budgets.

> Mechanism should be provisioned to track where the timing requirement is violated and by how much
I’ll leave the OAM for later. However there are already many high accuracy performance measurement techniques and protocols for this.
>> For this I mean something recorded in the same packet with the deadlines. If it misses the deadline, the receiver may need to know where it’s violated. Other independent methods are possible, but it’s better to consider if it can be integrated in the current proposal.

> such as PIPO and PIEO
I’ve heard of PIFO (Push In First out) but not PIPO. Is this a typo or something new?
I agree that there are mechanisms that are optimized for hardware, but I have come up with a very nice hardware implementation for PEDF
and prefer to find hardware implementations for optimal schedulers, rather than to determine schedulers based on optimal hardware.
>> Sorry that’s a typo. I mean PIFO (although we do have a paper under review using the name PIPO). Yes I agree those are just abstract primitives. The actual implementation, if customized to a particular algorithm, would be simpler.

Y(J)S

Hi Yaakov,

Just got a chance to read your draft. I agree with the comments of the others that this is a very interesting work. I’ll just add a few points.

Best regards,
Haoyu

All,

  *   DetNet is most relevant since the whole point is to control end-to-end latency of a time-sensitive flow.
  *   Spring is also directly relevant due to the use of a stack in the header and the combined approach just mentioned.
  *   PCE is relevant to the case of a central server jointly computing an optimal path and local deadline stack.
I’ll let the chairs decide where discussions should be held.

Y(J)S

---

From: Toerless Eckert <[email-masked]>
Date: Mon, 8 Mar 2021 17:36:23 +0100

> Yaakov wrote:
> I agree that there are mechanisms that are optimized for hardware, but I have come up with a very nice hardware implementation for PEDF  and prefer to find hardware implementations for optimal schedulers, rather than to determine schedulers based on optimal hardware.

PEDF ?

There was a long debate in the congestion control technology in TSV about the scalability
issues of flow-aware AQM mechanisms and AFAIK, ultimately, the AQM mechanisms that won out
where the ones that did not require this (such as PIE).  We also had round 1 of
deterministic services (if i may call them that) using rfc2212 with RSVP fail because
of real or assumed infeasibility to scale on multiple dimensions. IMHO it actually
failed also because of assumed infeasibility becasue there was no good analysis and
documentation of what actually could be made to work (badmouthing...).

>From this experience i can only recommend to make sure that we do understand what and
how something is feasible to be implemented a what scale and speed. 

For example, i am not aware to have seen general purpose EDF hardware at scale. But
i am very interested for any pointers. On the research side, this one is the
oldest one for PIFO i know:

http://web.mit.edu/pifo/pifo-sigcomm.pdf

[ i am of course mentioning PIFO because by using deadline as the PIFO rank, one has
  a simple approach to implement EDF].

In this work, if i remember my analysis correctly, the scale still depends on the
number of flows and the ability to identify packets to flows (need to read it again though).

This _may_ be acceptable in specific use-cases but should IMHO be well understood
and documented, especially when the view from the outside is that by using e.g.: SR
packet headers it is "looking" as if there is really no per-flow scaling aspect to
the hardware requiremens. After all, the idea of source routing with SR and having the
state in the packet is to eliminate the need to have) and scale it in the router.

Cheers
    Toerless

> _______________________________________________
> detnet mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/detnet

-- 
---
[email-masked]

---

From: Yaakov Stein <[email-masked]>
Date: Tue, 9 Mar 2021 04:34:19 +0000

Toerless

I describe PEDF in the ID. It is a twist on the combination of strict priority and EDF.

Basically there can be multiple time sensitive flows with different priorities and non-sensitive traffic as well.
With PEDF one first looks at the TS structure with the highest priority,
however, one doesn't take the packet in the highest priority (like in regular strict priority),
but rather you check whether its deadline is closer than the maximum packet clock-out time from now.
If not one checks the next highest priority, with full knowledge that one can send another packet and still make it back in time.

This can be extended to WFQ as well.

Using PEDF loosens the requirements on the optimization.
In fact, PEDF and the simple calculation on each flow separately
usually outperforms EDF and a more sophisticated optimization calculation.
(I don't yet have an analytical explanation of this...)

Y(J)S

-----Original Message-----
From: Toerless Eckert <[email-masked]> 
Sent: 08/03/2021 18:36
To: Yaakov Stein <[email-masked]>
Cc: [email-masked]; Haoyu Song <[email-masked]>; [email-masked]; [email-masked]
Subject: PEDF / PIFO / ... (was: Re: [Detnet] new draft on segment routing approach to TSN)

PEDF ?

There was a long debate in the congestion control technology in TSV about the scalability issues of flow-aware AQM mechanisms and AFAIK, ultimately, the AQM mechanisms that won out where the ones that did not require this (such as PIE).  We also had round 1 of deterministic services (if i may call them that) using rfc2212 with RSVP fail because of real or assumed infeasibility to scale on multiple dimensions. IMHO it actually failed also because of assumed infeasibility becasue there was no good analysis and documentation of what actually could be made to work (badmouthing...).

>From this experience i can only recommend to make sure that we do understand what and how something is feasible to be implemented a what scale and speed. 

For example, i am not aware to have seen general purpose EDF hardware at scale. But i am very interested for any pointers. On the research side, this one is the oldest one for PIFO i know:

https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fweb.mit.edu%2Fpifo%2Fpifo-sigcomm.pdf&amp;data=04%7C01%7Cyaakov_s%40rad.com%7Ccb4381151b5f494cc2ce08d8e2505295%7Cf9047108cc2c4e4897a343fad1b3bf9d%7C1%7C0%7C637508181910510455%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=LCK5i%2BmLewtZdjJIK7RlDPFqIRONvgefZxaX1f91ckw%3D&amp;reserved=0

In this work, if i remember my analysis correctly, the scale still depends on the number of flows and the ability to identify packets to flows (need to read it again though).

This _may_ be acceptable in specific use-cases but should IMHO be well understood and documented, especially when the view from the outside is that by using e.g.: SR packet headers it is "looking" as if there is really no per-flow scaling aspect to the hardware requiremens. After all, the idea of source routing with SR and having the state in the packet is to eliminate the need to have) and scale it in the router.

Cheers
    Toerless

> From: Haoyu Song 
> <[email-masked]<mailto:[email-masked]>>
> Sent: 05/03/2021 22:46
> To: Yaakov Stein <[email-masked]<mailto:[email-masked]>>; 
> [email-masked]<mailto:[email-masked]>; 
> [email-masked]<mailto:[email-masked]>; 
> [email-masked]<mailto:[email-masked]>
> To: [email-masked]<mailto:[email-masked]>; 
> [email-masked]<mailto:[email-masked]>; 
> [email-masked]<mailto:[email-masked]>
> I would like to call your attention to a new ID 
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.
> ietf.org%2Farchive%2Fid%2Fdraft-stein-srtsn-00.txt&amp;data=04%7C01%7C
> yaakov_s%40rad.com%7Ccb4381151b5f494cc2ce08d8e2505295%7Cf9047108cc2c4e
> 4897a343fad1b3bf9d%7C1%7C0%7C637508181910510455%7CUnknown%7CTWFpbGZsb3
> d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7
> C1000&amp;sdata=KhdBd9Z2XCTVUvweatNWyCgwZDVKssHJF2W%2Fz41Q1uY%3D&amp;r
> eserved=0<https://eur01.safelinks.protection.outlook.com/?url=https%3A
> %2F%2Fwww.ietf.org%2Farchive%2Fid%2Fdraft-stein-srtsn-00.txt&amp;data=
> 04%7C01%7Cyaakov_s%40rad.com%7Ccb4381151b5f494cc2ce08d8e2505295%7Cf904
> 7108cc2c4e4897a343fad1b3bf9d%7C1%7C0%7C637508181910510455%7CUnknown%7C
> TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVC
> I6Mn0%3D%7C1000&amp;sdata=KhdBd9Z2XCTVUvweatNWyCgwZDVKssHJF2W%2Fz41Q1u
> Y%3D&amp;reserved=0> which describes using a stack-based approach 
> (similar to segment routing) to time sensitive networking.
> It furthermore proposes combining segment routing with this approach 
> to TSN resulting in a unified approach to forwarding and scheduling.
> 
> Apologies for simultaneously sending to 3 lists, but I am not sure 
> which WG is the most appropriate for discussions of this topic.
> 

> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.
> ietf.org%2Fmailman%2Flistinfo%2Fdetnet&amp;data=04%7C01%7Cyaakov_s%40r
> ad.com%7Ccb4381151b5f494cc2ce08d8e2505295%7Cf9047108cc2c4e4897a343fad1
> b3bf9d%7C1%7C0%7C637508181910510455%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC
> 4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sd
> ata=v3Dh%2Bq0%2F8eYutLOjnkz0Aa%2BvieQW8CoRhhbv7JfN77c%3D&amp;reserved=
> 0

--
---
[email-masked]

---

From: Balázs Varga A <[email-masked]>
Date: Tue, 9 Mar 2021 09:17:25 +0000

Hi Bryan,

Sure, but there is no free lunch. :--))
You have added the problem of calculation of the budget.
Knowing your exact egress time in advance is not an easy task.
There will be an inaccuracy in budget calculations.

Cheers
Bala'zs

From: detnet <[email-masked]> On Behalf Of Liubingyang (Bryan)
Sent: Monday, March 8, 2021 3:31 PM
To: Yaakov Stein <[email-masked]>; Haoyu Song <[email-masked]>; detnet <[email-masked]>; spring <[email-masked]>; pce <[email-masked]>
Subject: Re: [Detnet] new draft on segment routing approach to TSN

Some advantages.

Just my two cents.

Bryan

From:Yaakov Stein <[email-masked]<mailto:[email-masked]>>
To:Liubingyang (Bryan) <[email-masked]<mailto:[email-masked]>>;Haoyu Song <[email-masked]<mailto:[email-masked]>>;detnet <[email-masked]<mailto:[email-masked]>>;spring <[email-masked]<mailto:[email-masked]>>;pce <[email-masked]<mailto:[email-masked]>>
Date:2021-03-08 21:38:20
Subject:RE: new draft on segment routing approach to TSN

Bryan

Y(J)S

From: Liubingyang (Bryan) <[email-masked]<mailto:[email-masked]>>
Sent: 08/03/2021 14:55
To: Yaakov Stein <[email-masked]<mailto:[email-masked]>>; Haoyu Song <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: RE: new draft on segment routing approach to TSN

Hi Yaakov,

Bryan (Bingyang Liu)

Haoyu

Y(J)S

Hi Yaakov,

Some feedback inline.

Best regards,
Haoyu

Haoyu,

Y(J)S

Best regards,
Haoyu

All,

I would like to call your attention to a new ID https://www.ietf.org/archive/id/draft-stein-srtsn-00.txt<https://protect2.fireeye.com/v1/url?k=bb6c9c54-e4f7a554-bb6cdccf-86fc6812c361-ade234225c5dc3b6&q=1&e=61fd3663-a974-4041-8021-790fa60fae0d&u=https%3A%2F%2Feur01.safelinks.protection.outlook.com%2F%3Furl%3Dhttps%253A%252F%252Fwww.ietf.org%252Farchive%252Fid%252Fdraft-stein-srtsn-00.txt%26data%3D04%257C01%257Cyaakov_s%2540rad.com%257C71c93afad8614e5d212e08d8e23156c8%257Cf9047108cc2c4e4897a343fad1b3bf9d%257C1%257C0%257C637508048843821932%257CUnknown%257CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%253D%257C1000%26sdata%3DJ5d%252Bz%252BmTnTm3C7%252FfD0YSo%252FXMk%252F67YxHlYVRfvv%252Fthcg%253D%26reserved%3D0>
which describes using a stack-based approach (similar to segment routing) to time sensitive networking.
It furthermore proposes combining segment routing with this approach to TSN
resulting in a unified approach to forwarding and scheduling.

Y(J)S

---

From: "Liubingyang (Bryan)" <[email-masked]>
Date: Tue, 9 Mar 2021 10:13:55 +0000

You are absolutely right : )

I see the remaining budget calculation as the difficulty in device implementation.
But I see the bit length as protocol overhead, and time sync as deployment difficulty.

Actually all possibilities can be discussed, and when used in different scenarios, different profiles (combinations of modules) may be used.

Best
Bryan

From: Balázs Varga A [mailto:[email-masked]]
Sent: Tuesday, March 9, 2021 5:17 PM
To: Liubingyang (Bryan) <[email-masked]>; Yaakov Stein <[email-masked]>; Haoyu Song <[email-masked]>; detnet <[email-masked]>; spring <[email-masked]>; pce <[email-masked]>
Subject: RE: new draft on segment routing approach to TSN

Hi Bryan,

Cheers
Bala'zs

From: detnet <[email-masked]<mailto:[email-masked]>> On Behalf Of Liubingyang (Bryan)
Sent: Monday, March 8, 2021 3:31 PM
To: Yaakov Stein <[email-masked]<mailto:[email-masked]>>; Haoyu Song <[email-masked]<mailto:[email-masked]>>; detnet <[email-masked]<mailto:[email-masked]>>; spring <[email-masked]<mailto:[email-masked]>>; pce <[email-masked]<mailto:[email-masked]>>
Subject: Re: [Detnet] new draft on segment routing approach to TSN

Some advantages.

Just my two cents.

Bryan

Bryan

Y(J)S

Hi Yaakov,

Bryan (Bingyang Liu)

Haoyu

Y(J)S

Hi Yaakov,

Some feedback inline.

Best regards,
Haoyu

Haoyu,

Y(J)S

Hi Yaakov,

Best regards,
Haoyu

All,

Y(J)S