Subject: [Pce] new draft on segment routing approach to TSN

From: Yaakov Stein <[email-masked]>
Date: Tue, 23 Feb 2021 13:13:44 +0000

All,

I would like to call your attention to a new ID https://www.ietf.org/archive/id/draft-stein-srtsn-00.txt
which describes using a stack-based approach (similar to segment routing) to time sensitive networking.
It furthermore proposes combining segment routing with this approach to TSN
resulting in a unified approach to forwarding and scheduling.

The draft is information at this point, since it discusses the concepts and does not yet pin down the precise formats.

Apologies for simultaneously sending to 3 lists,
but I am not sure which WG is the most appropriate for discussions of this topic.

  *   DetNet is most relevant since the whole point is to control end-to-end latency of a time-sensitive flow.
  *   Spring is also directly relevant due to the use of a stack in the header and the combined approach just mentioned.
  *   PCE is relevant to the case of a central server jointly computing an optimal path and local deadline stack.
I'll let the chairs decide where discussions should be held.

Y(J)S

---

From: "UTTARO, JAMES" <[email-masked]>
Date: Wed, 24 Feb 2021 14:42:40 +0000

Very interesting approach.

Thanks,
              Jim Uttaro

From: spring <[email-masked]> On Behalf Of Yaakov Stein
Sent: Tuesday, February 23, 2021 8:14 AM
To: [email-masked]; [email-masked]; [email-masked]
Subject: [spring] new draft on segment routing approach to TSN

All,

I would like to call your attention to a new ID https://www.ietf.org/archive/id/draft-stein-srtsn-00.txt<https://urldefense.com/v3/__https:/www.ietf.org/archive/id/draft-stein-srtsn-00.txt__;!!BhdT!ywXwCcS1Fp_Gs_t6keSpxFAOxace4nmdTFtKP9kkK9fmKk2gtwSZz2d6RFs$>
which describes using a stack-based approach (similar to segment routing) to time sensitive networking.
It furthermore proposes combining segment routing with this approach to TSN
resulting in a unified approach to forwarding and scheduling.

Y(J)S

---

From: Tianran Zhou <[email-masked]>
Date: Thu, 25 Feb 2021 08:13:43 +0000

Hi Yaakov,

This is an interesting topic.
After a quick review, there are several questions as follows:
1. It's clear to me to have a deadline for each packet. So that router can schedule the packet based on the urgency. But what's the motivation to split the end to end deadline to several local ones?
2. How to divide an end to end deadline into several local deadlines? Is there any example algorithm that could be used by the controller?
3. As far as I know, most devices do not support edf. I am not sure whether your proposal based on edf could really be useful.

Cheers,
Tianran

From: Pce [mailto:[email-masked]] On Behalf Of Yaakov Stein
Sent: Tuesday, February 23, 2021 9:14 PM
To: [email-masked]; [email-masked]; [email-masked]
Subject: [Pce] new draft on segment routing approach to TSN

All,

Y(J)S

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Thu, 25 Feb 2021 17:40:29 +0000

Hi Yaakov and all:

Whereever Yaakov decides to place it I'll be there supporting the work. The draft itself is incredibly well-written and information-rich.
Note that there's also work in RAW that mentions SR operation DetNet related operations (draft-pthubert-raw-architecture<https://tools.ietf.org/html/draft-pthubert-raw-architecture-05>). RAW has vested interest in intelligent forwarding decision, that would be the trademark vs. DetNet. With this draft, the forwarding is not based on Qbv schedule but the forwarder has some latitude as long as it matches the hop deadline. So RAW may be a good place.
And then there's draft-chen-detnet-sr-based-bounded-latency<https://tools.ietf.org/html/draft-chen-detnet-sr-based-bounded-latency-01>. Ideally all these related items would progress in the same room.

Also a few notes on the draft itself:
- maybe use latency instead of delay; it would be nice to maybe define delay as something else, e.g., the delay representing the time the packet spends queued in one hop vs. the latency that is end to end?
- not sure the term green wave is well understood by the public here; the draft gives the impression that the TSN path is faster than the best effort and involves no queueing. For the most part that is untrue; the latency is bounded but for most flows it is longer than best effort. Best effort can be really fast with passthrough in an empty network. The problem is the long tail and possibly congestion loss. For TSN, there can be very special flows that will traverse the city with all the lights green, but usually there'll be queuing. The difference is that the queueing latency is constant and the overall latency is withing bounds.
- Time triggered is not the only TSN operation. I wonder what the draft would become with asynchronous shaper in mind. We designed (and as I must announce, patented as US9602420<https://patents.google.com/patent/US9602420>) a system very similar to the one proposed in the draft, but that is designed to adapt QoS depending on whether the packet is early or late vs. its schedule, and not tagging the schedule in the since the latency is considered end to end not hop by hop. The use case is slightly different since we apply this without a global controller and a provable guarantees all flows will meet the deadline - so not really detnet-, but more like a best effort that all flows meet their deadline in a stochastic environment. If Yaakov is interested, we can contribute on that aspect.

Good luck with the draft,

Pascal

From: detnet <[email-masked]> On Behalf Of Tianran Zhou
Sent: jeudi 25 février 2021 9:14
To: Yaakov Stein <[email-masked]>; [email-masked]; [email-masked]; [email-masked]
Subject: Re: [Detnet] new draft on segment routing approach to TSN

Hi Yaakov,

Cheers,
Tianran

From: Pce [mailto:[email-masked]] On Behalf Of Yaakov Stein
Sent: Tuesday, February 23, 2021 9:14 PM
To: [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: [Pce] new draft on segment routing approach to TSN

All,

Y(J)S

---

From: Yaakov Stein <[email-masked]>
Date: Thu, 25 Feb 2021 20:33:05 +0000

Pascal,

Wow, there is a lot to respond to.

Thanks for the pointers. I was not aware of this work.
However, I believe this proposal is both different and can be complementary to the ones you mention.

I was going to add a section on how to reproduce Qbv behavior using a stack of deadlines (the argument is a bit tricky, but straightforward).
I don't necessarily think that is a good idea, but it should be there to counter claims that SRTSN only reduces the expectation
but can't provide an absolute upper bound on latency (when based on EDF).

I like to use the term latency, but sometimes use delay to avoid overusing it.

I thought that "green wave" would be understood by those who drive cars in cities where traffic light coordination is common.

I attempted to describe a data structure for EDF to be used instead of a queue,
but never implied that "queueing" (meaning packets waiting to be transmitted) doesn't occur.
If that were the case there would not be a need for any data structure at all.
(Another thing I didn't get into was the trade-offs among the different options,
 from worst case complexity and difficulty to implement in hardware PoVs.)

I took a quick look at US 9602420 and will study it further.
However the purpose of this draft is not to mandate any specific scheduling mechanism,
rather to show how such mechanisms can be integrated and hopefully more readily distributed.
Incidentally I am working on a joint path+deadline optimization algorithm which only requires partial information.

Proofs of absolute upper bounds on latency are nice, but generally come at the price of either

  1.  significantly reduced bandwidth efficiency (like TDM, or equivalently time gating as in Qbv or FlexE),
  2.  unscalable computation
  3.  relatively loose deadlines to being with (like Andrews Zhang)
  4.  assumptions that aren't necessarily obeyed in real life (like a lot of the network calculus proofs).
I would be happy to discuss your variant to see in which class it falls.

Y(J)S

From: Pascal Thubert (pthubert) <[email-masked]>
Sent: 25/02/2021 19:40
To: Tianran Zhou <[email-masked]>; Yaakov Stein <[email-masked]>; [email-masked]; [email-masked]; [email-masked]
Subject: RE: new draft on segment routing approach to TSN

CAUTION: External sender. Do not click links or open attachments unless you know the content is safe.
Hi Yaakov and all:

Whereever Yaakov decides to place it I'll be there supporting the work. The draft itself is incredibly well-written and information-rich.
Note that there's also work in RAW that mentions SR operation DetNet related operations (draft-pthubert-raw-architecture<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Fdraft-pthubert-raw-architecture-05&data=04%7C01%7Cyaakov_s%40rad.com%7C5a6ae47118cf453d263d08d8d9b4776b%7Cf9047108cc2c4e4897a343fad1b3bf9d%7C1%7C0%7C637498716423474214%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=ml5Lr%2FvJHvhBtJnP6KqQ2gE20rkYMihRA9NYOEhye4E%3D&reserved=0>). RAW has vested interest in intelligent forwarding decision, that would be the trademark vs. DetNet. With this draft, the forwarding is not based on Qbv schedule but the forwarder has some latitude as long as it matches the hop deadline. So RAW may be a good place.
And then there's draft-chen-detnet-sr-based-bounded-latency<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Fdraft-chen-detnet-sr-based-bounded-latency-01&data=04%7C01%7Cyaakov_s%40rad.com%7C5a6ae47118cf453d263d08d8d9b4776b%7Cf9047108cc2c4e4897a343fad1b3bf9d%7C1%7C0%7C637498716423484210%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=7DTzliy8OHRPIqZ719W7Ul2lPqgpcxcOU4oYb8VmSSM%3D&reserved=0>. Ideally all these related items would progress in the same room.

Also a few notes on the draft itself:
- maybe use latency instead of delay; it would be nice to maybe define delay as something else, e.g., the delay representing the time the packet spends queued in one hop vs. the latency that is end to end?
- not sure the term green wave is well understood by the public here; the draft gives the impression that the TSN path is faster than the best effort and involves no queueing. For the most part that is untrue; the latency is bounded but for most flows it is longer than best effort. Best effort can be really fast with passthrough in an empty network. The problem is the long tail and possibly congestion loss. For TSN, there can be very special flows that will traverse the city with all the lights green, but usually there'll be queuing. The difference is that the queueing latency is constant and the overall latency is withing bounds.
- Time triggered is not the only TSN operation. I wonder what the draft would become with asynchronous shaper in mind. We designed (and as I must announce, patented as US9602420<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatents.google.com%2Fpatent%2FUS9602420&data=04%7C01%7Cyaakov_s%40rad.com%7C5a6ae47118cf453d263d08d8d9b4776b%7Cf9047108cc2c4e4897a343fad1b3bf9d%7C1%7C0%7C637498716423494204%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=aSHLwTc0xl9SbP%2BTLsd4R6EY542%2FOSII1bfC3Jrpueg%3D&reserved=0>) a system very similar to the one proposed in the draft, but that is designed to adapt QoS depending on whether the packet is early or late vs. its schedule, and not tagging the schedule in the since the latency is considered end to end not hop by hop. The use case is slightly different since we apply this without a global controller and a provable guarantees all flows will meet the deadline - so not really detnet-, but more like a best effort that all flows meet their deadline in a stochastic environment. If Yaakov is interested, we can contribute on that aspect.

Pascal

From: detnet <[email-masked]<mailto:[email-masked]>> On Behalf Of Tianran Zhou
Sent: jeudi 25 février 2021 9:14
To: Yaakov Stein <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: Re: [Detnet] new draft on segment routing approach to TSN

Hi Yaakov,

Cheers,
Tianran

All,

I would like to call your attention to a new ID https://www.ietf.org/archive/id/draft-stein-srtsn-00.txt<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ietf.org%2Farchive%2Fid%2Fdraft-stein-srtsn-00.txt&data=04%7C01%7Cyaakov_s%40rad.com%7C5a6ae47118cf453d263d08d8d9b4776b%7Cf9047108cc2c4e4897a343fad1b3bf9d%7C1%7C0%7C637498716423494204%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=8JOBA97tVXOHG3Ao6HOcXU%2FkaqYcMyIjVXQfVRA4twU%3D&reserved=0>
which describes using a stack-based approach (similar to segment routing) to time sensitive networking.
It furthermore proposes combining segment routing with this approach to TSN
resulting in a unified approach to forwarding and scheduling.

Y(J)S

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Thu, 25 Feb 2021 21:26:57 +0000

Hello Yaakov

Please see below:

Le 25 févr. 2021 à 21:33, Yaakov Stein <[email-masked]> a écrit :

Pascal,

Indeed; the questions that this raises are:

- which WG, so I added RAW to the candidate list in this thread

- how to package the complementary proposals in how many draft

Your excellent work focuses on SR signaling in the forwarding plane with strict intermediate latencies at each stage. It would be grand to neatly insert this in a SRv6 together with other information that the RAW PSE processes in its forwarding/scheduling decision.

There are multiple dimensions in which the scope of your work could be enlarged; one of them would be to use a similar signaling but with a best effort objective, which is what I discussed. This could easily be combined if we decide to, so the same signaling serves both use cases. And maybe more, to be discussed.

I was going to add a section on how to reproduce Qbv behavior using a stack of deadlines (the argument is a bit tricky, but straightforward).
I don’t necessarily think that is a good idea, but it should be there to counter claims that SRTSN only reduces the expectation
but can’t provide an absolute upper bound on latency (when based on EDF).

Very cool;

Your decision, at least as long as it is personal submission ; but having a clear terminology is useful. Literature value is of lesser importance here I guess. Latency is what we use in DetNet and RAW. Delay often refers to extra, possibly unwanted time, like a queueing delay. At least you’d make my reader life easier by defining your terminology and sticking to it quite strictly : )

I thought that “green wave” would be understood by those who drive cars in cities where traffic light coordination is common.

Yes, and I think you were clear enough; but then only a few deterministic flows really operate like that. Another flow that use the same outgoing port will have to suffer some delay in a time-triggered queue till the port is free, the delay being taken from the end to end latency budget. My suggestion was to reconsider the text to avoid giving the impression that all flows benefit from that green wave.

Neat

but never implied that “queueing” (meaning packets waiting to be transmitted) doesn’t occur.
If that were the case there would not be a need for any data structure at all.

Cool

(Another thing I didn’t get into was the trade-offs among the different options,
 from worst case complexity and difficulty to implement in hardware PoVs.)

I’d love to read that too

It’s not as deep as an academic paper but a good hint at how you can turn schedule into QoS or another technique to reorder the set of packets that compete for the port.

Usually you’re expected to provide the meaning of the signaling and the visible behavior of the node; the way the node implements the does not need to be spec but some hints on how that can be achieved are welcome, possibly in annex.

I’d love to chat in this; missing the face to face meetings! Could it be that we do SF in summer?

This is exactly why we did what we did; it also comes at the cost of gathering very specific info on the flows and not all flows can do that, think CBR.

I need to read it, sorry I was not aware enough of that line of work.

  1.
  2.  assumptions that aren’t necessarily obeyed in real life (like a lot of the network calculus proofs).
I would be happy to discuss your variant to see in which class it falls.

Same here!

Pascal

Y(J)S

Whereever Yaakov decides to place it I’ll be there supporting the work. The draft itself is incredibly well-written and information-rich.
Note that there’s also work in RAW that mentions SR operation DetNet related operations (draft-pthubert-raw-architecture<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Fdraft-pthubert-raw-architecture-05&data=04%7C01%7Cyaakov_s%40rad.com%7C5a6ae47118cf453d263d08d8d9b4776b%7Cf9047108cc2c4e4897a343fad1b3bf9d%7C1%7C0%7C637498716423474214%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=ml5Lr%2FvJHvhBtJnP6KqQ2gE20rkYMihRA9NYOEhye4E%3D&reserved=0>). RAW has vested interest in intelligent forwarding decision, that would be the trademark vs. DetNet. With this draft, the forwarding is not based on Qbv schedule but the forwarder has some latitude as long as it matches the hop deadline. So RAW may be a good place.
And then there’s draft-chen-detnet-sr-based-bounded-latency<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Fdraft-chen-detnet-sr-based-bounded-latency-01&data=04%7C01%7Cyaakov_s%40rad.com%7C5a6ae47118cf453d263d08d8d9b4776b%7Cf9047108cc2c4e4897a343fad1b3bf9d%7C1%7C0%7C637498716423484210%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=7DTzliy8OHRPIqZ719W7Ul2lPqgpcxcOU4oYb8VmSSM%3D&reserved=0>. Ideally all these related items would progress in the same room.

Also a few notes on the draft itself:
- maybe use latency instead of delay; it would be nice to maybe define delay as something else, e.g., the delay representing the time the packet spends queued in one hop vs. the latency that is end to end?
- not sure the term green wave is well understood by the public here; the draft gives the impression that the TSN path is faster than the best effort and involves no queueing. For the most part that is untrue; the latency is bounded but for most flows it is longer than best effort. Best effort can be really fast with passthrough in an empty network. The problem is the long tail and possibly congestion loss. For TSN, there can be very special flows that will traverse the city with all the lights green, but usually there’ll be queuing. The difference is that the queueing latency is constant and the overall latency is withing bounds.
- Time triggered is not the only TSN operation. I wonder what the draft would become with asynchronous shaper in mind. We designed (and as I must announce, patented as US9602420<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatents.google.com%2Fpatent%2FUS9602420&data=04%7C01%7Cyaakov_s%40rad.com%7C5a6ae47118cf453d263d08d8d9b4776b%7Cf9047108cc2c4e4897a343fad1b3bf9d%7C1%7C0%7C637498716423494204%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=aSHLwTc0xl9SbP%2BTLsd4R6EY542%2FOSII1bfC3Jrpueg%3D&reserved=0>) a system very similar to the one proposed in the draft, but that is designed to adapt QoS depending on whether the packet is early or late vs. its schedule, and not tagging the schedule in the since the latency is considered end to end not hop by hop. The use case is slightly different since we apply this without a global controller and a provable guarantees all flows will meet the deadline – so not really detnet-, but more like a best effort that all flows meet their deadline in a stochastic environment. If Yaakov is interested, we can contribute on that aspect.

Pascal

Hi Yaakov,

1. It’s clear to me to have a deadline for each packet. So that router can schedule the packet based on the urgency. But what’s the motivation to split the end to end deadline to several local ones?

Cheers,
Tianran

All,

  *   DetNet is most relevant since the whole point is to control end-to-end latency of a time-sensitive flow.
  *   Spring is also directly relevant due to the use of a stack in the header and the combined approach just mentioned.
  *   PCE is relevant to the case of a central server jointly computing an optimal path and local deadline stack.
I’ll let the chairs decide where discussions should be held.

Y(J)S

---

From: "Jakob Heitz (jheitz)" <[email-masked]>
Date: Thu, 25 Feb 2021 23:35:34 +0000

Another thing to consider is packets dropped due to bit errors.
Possibilities: Link level acknowledgement and preemptive retransmission
if the link would otherwise be idle. I.e. "just in case" retransmission.

Regards,
Jakob.

From: spring <[email-masked]> On Behalf Of Pascal Thubert (pthubert)
Sent: Thursday, February 25, 2021 1:27 PM
To: Yaakov Stein <[email-masked]>
Cc: [email-masked]; Tianran Zhou <[email-masked]>; [email-masked]; [email-masked]
Subject: Re: [spring] new draft on segment routing approach to TSN

Hello Yaakov

Please see below:

Le 25 févr. 2021 à 21:33, Yaakov Stein <[email-masked]<mailto:[email-masked]>> a écrit :

Pascal,

Very cool;

Neat

Cool

  1.

Same here!

Pascal

Y(J)S

From: Pascal Thubert (pthubert) <[email-masked]<mailto:[email-masked]>>
Sent: 25/02/2021 19:40
To: Tianran Zhou <[email-masked]<mailto:[email-masked]>>; Yaakov Stein <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: RE: new draft on segment routing approach to TSN

Pascal

Hi Yaakov,

Cheers,
Tianran

All,

Y(J)S

---

From: Yaakov Stein <[email-masked]>
Date: Fri, 26 Feb 2021 04:31:49 +0000

Pascal,

Please accept my apologies for having misunderstood your intention regarding “green waves”.

I originally read it as two separate statements, namely

  1.  people won’t understand the “green wave” analogy
  2.  people in general think that TSN is magic and no packet ever waits
while I now understand that you intended one statement

  1.  people will misunderstand the analogy to green waves and believe that I am implying that no packet ever waits.

My reply to your true intention is that even in those places where traffic lights are coordinated
it doesn’t mean that every car is first to go through every light or that you don’t have to slow down before lights.
It merely implies that there is high correlation between car arrival times and green light periods.
This is what Qbv and calendaring are supposed to accomplish,
and if people will be misled then perhaps I need to find another analogy.

Perhaps something with bus arrival times is more appropriate where people queue up to enter the bus.
The reason I wanted to avoid that one is that buses don’t stick to schedule.
Since they wait longer when there happen to be more passengers getting on,
they then meet even more passengers at the next stop, while the following bus meets fewer, resulting in bunching.

Y(J)S

From: Pascal Thubert (pthubert) <[email-masked]>
Sent: 25/02/2021 23:27
To: Yaakov Stein <[email-masked]>
Cc: Tianran Zhou <[email-masked]>; [email-masked]; [email-masked]; [email-masked]
Subject: Re: new draft on segment routing approach to TSN

Hello Yaakov

Please see below:

Pascal,

Very cool;

Neat

Cool

  1.

Same here!

Pascal

Y(J)S

Whereever Yaakov decides to place it I’ll be there supporting the work. The draft itself is incredibly well-written and information-rich.
Note that there’s also work in RAW that mentions SR operation DetNet related operations (draft-pthubert-raw-architecture<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Fdraft-pthubert-raw-architecture-05&data=04%7C01%7Cyaakov_s%40rad.com%7C0f58b1362ad8426a72fc08d8d9d41ec1%7Cf9047108cc2c4e4897a343fad1b3bf9d%7C1%7C0%7C637498852383325595%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=%2FVosVlYVBfSoHawDj8o6MnU%2B8boPbF7b90oC982lsm4%3D&reserved=0>). RAW has vested interest in intelligent forwarding decision, that would be the trademark vs. DetNet. With this draft, the forwarding is not based on Qbv schedule but the forwarder has some latitude as long as it matches the hop deadline. So RAW may be a good place.
And then there’s draft-chen-detnet-sr-based-bounded-latency<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Fdraft-chen-detnet-sr-based-bounded-latency-01&data=04%7C01%7Cyaakov_s%40rad.com%7C0f58b1362ad8426a72fc08d8d9d41ec1%7Cf9047108cc2c4e4897a343fad1b3bf9d%7C1%7C0%7C637498852383335588%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=6pvuDE5KNHlhm%2Bu8Tj3z72EyR343MKzO8aEuLp%2BOiu0%3D&reserved=0>. Ideally all these related items would progress in the same room.

Also a few notes on the draft itself:
- maybe use latency instead of delay; it would be nice to maybe define delay as something else, e.g., the delay representing the time the packet spends queued in one hop vs. the latency that is end to end?
- not sure the term green wave is well understood by the public here; the draft gives the impression that the TSN path is faster than the best effort and involves no queueing. For the most part that is untrue; the latency is bounded but for most flows it is longer than best effort. Best effort can be really fast with passthrough in an empty network. The problem is the long tail and possibly congestion loss. For TSN, there can be very special flows that will traverse the city with all the lights green, but usually there’ll be queuing. The difference is that the queueing latency is constant and the overall latency is withing bounds.
- Time triggered is not the only TSN operation. I wonder what the draft would become with asynchronous shaper in mind. We designed (and as I must announce, patented as US9602420<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatents.google.com%2Fpatent%2FUS9602420&data=04%7C01%7Cyaakov_s%40rad.com%7C0f58b1362ad8426a72fc08d8d9d41ec1%7Cf9047108cc2c4e4897a343fad1b3bf9d%7C1%7C0%7C637498852383345583%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=9gwFvvZITbWsRvAA12JmP1zzTwyLuI9DVA9ph2fuEH4%3D&reserved=0>) a system very similar to the one proposed in the draft, but that is designed to adapt QoS depending on whether the packet is early or late vs. its schedule, and not tagging the schedule in the since the latency is considered end to end not hop by hop. The use case is slightly different since we apply this without a global controller and a provable guarantees all flows will meet the deadline – so not really detnet-, but more like a best effort that all flows meet their deadline in a stochastic environment. If Yaakov is interested, we can contribute on that aspect.

Pascal

Hi Yaakov,

Cheers,
Tianran

All,

I would like to call your attention to a new ID https://www.ietf.org/archive/id/draft-stein-srtsn-00.txt<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ietf.org%2Farchive%2Fid%2Fdraft-stein-srtsn-00.txt&data=04%7C01%7Cyaakov_s%40rad.com%7C0f58b1362ad8426a72fc08d8d9d41ec1%7Cf9047108cc2c4e4897a343fad1b3bf9d%7C1%7C0%7C637498852383345583%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=xs2g3OnkF18VI6ZFHO5LIg5EmJW0m%2BLUHzflSq0I1DY%3D&reserved=0>
which describes using a stack-based approach (similar to segment routing) to time sensitive networking.
It furthermore proposes combining segment routing with this approach to TSN
resulting in a unified approach to forwarding and scheduling.

Y(J)S

---

From: Yaakov Stein <[email-masked]>
Date: Fri, 26 Feb 2021 04:39:56 +0000

Jakob

Yes, one always needs to consider bit errors (even if they are very rare in wireline nowadays).

I’m just not sure where you want to hang this on the present discussion.

A local retransmission mechanism such as the one you suggest
would obviously thwart any calendaring mechanism that attempts to coordinate scheduling based on traffic timing
(unless the detection, signaling, and retransmission takes place much faster than an opening time).
LIS could be made to work by retransmitting with the original timestamp, and EDF would work just fine.

Y(J)S

From: Jakob Heitz (jheitz) <[email-masked]>
Sent: 26/02/2021 01:36
To: Pascal Thubert (pthubert) <pthubert=[email-masked]>; Yaakov Stein <[email-masked]>
Cc: [email-masked]; Tianran Zhou <[email-masked]>; [email-masked]; [email-masked]
Subject: RE: new draft on segment routing approach to TSN

Regards,
Jakob.

From: spring <[email-masked]<mailto:[email-masked]>> On Behalf Of Pascal Thubert (pthubert)
Sent: Thursday, February 25, 2021 1:27 PM
To: Yaakov Stein <[email-masked]<mailto:[email-masked]>>
Cc: [email-masked]<mailto:[email-masked]>; Tianran Zhou <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: Re: [spring] new draft on segment routing approach to TSN

Hello Yaakov

Please see below:

Pascal,

Very cool;

Neat

Cool

  1.

Same here!

Pascal

Y(J)S

Whereever Yaakov decides to place it I’ll be there supporting the work. The draft itself is incredibly well-written and information-rich.
Note that there’s also work in RAW that mentions SR operation DetNet related operations (draft-pthubert-raw-architecture<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Fdraft-pthubert-raw-architecture-05&data=04%7C01%7Cyaakov_s%40rad.com%7C2a58f0004f474b0bfe0208d8d9e60f15%7Cf9047108cc2c4e4897a343fad1b3bf9d%7C1%7C0%7C637498929427225046%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=P467CZJoyNbw3wabSxpyO52vu3xCjpBdo1y582YRU3c%3D&reserved=0>). RAW has vested interest in intelligent forwarding decision, that would be the trademark vs. DetNet. With this draft, the forwarding is not based on Qbv schedule but the forwarder has some latitude as long as it matches the hop deadline. So RAW may be a good place.
And then there’s draft-chen-detnet-sr-based-bounded-latency<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Fdraft-chen-detnet-sr-based-bounded-latency-01&data=04%7C01%7Cyaakov_s%40rad.com%7C2a58f0004f474b0bfe0208d8d9e60f15%7Cf9047108cc2c4e4897a343fad1b3bf9d%7C1%7C0%7C637498929427235040%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=dwpzo8ihBfCIkWWLmItGm2AWkfi4uT340pjW7A4fni0%3D&reserved=0>. Ideally all these related items would progress in the same room.

Also a few notes on the draft itself:
- maybe use latency instead of delay; it would be nice to maybe define delay as something else, e.g., the delay representing the time the packet spends queued in one hop vs. the latency that is end to end?
- not sure the term green wave is well understood by the public here; the draft gives the impression that the TSN path is faster than the best effort and involves no queueing. For the most part that is untrue; the latency is bounded but for most flows it is longer than best effort. Best effort can be really fast with passthrough in an empty network. The problem is the long tail and possibly congestion loss. For TSN, there can be very special flows that will traverse the city with all the lights green, but usually there’ll be queuing. The difference is that the queueing latency is constant and the overall latency is withing bounds.
- Time triggered is not the only TSN operation. I wonder what the draft would become with asynchronous shaper in mind. We designed (and as I must announce, patented as US9602420<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatents.google.com%2Fpatent%2FUS9602420&data=04%7C01%7Cyaakov_s%40rad.com%7C2a58f0004f474b0bfe0208d8d9e60f15%7Cf9047108cc2c4e4897a343fad1b3bf9d%7C1%7C0%7C637498929427245036%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=AhjNR%2FD0U9RIg%2BprS1WN5nsC7FQzcVS7WSaIjCgDpyM%3D&reserved=0>) a system very similar to the one proposed in the draft, but that is designed to adapt QoS depending on whether the packet is early or late vs. its schedule, and not tagging the schedule in the since the latency is considered end to end not hop by hop. The use case is slightly different since we apply this without a global controller and a provable guarantees all flows will meet the deadline – so not really detnet-, but more like a best effort that all flows meet their deadline in a stochastic environment. If Yaakov is interested, we can contribute on that aspect.

Pascal

Hi Yaakov,

Cheers,
Tianran

All,

I would like to call your attention to a new ID https://www.ietf.org/archive/id/draft-stein-srtsn-00.txt<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ietf.org%2Farchive%2Fid%2Fdraft-stein-srtsn-00.txt&data=04%7C01%7Cyaakov_s%40rad.com%7C2a58f0004f474b0bfe0208d8d9e60f15%7Cf9047108cc2c4e4897a343fad1b3bf9d%7C1%7C0%7C637498929427245036%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=Y%2Bk%2B3NkrdcTFReuRbw%2FLxO0qLYrGBcMCDNfnd3R9Q4I%3D&reserved=0>
which describes using a stack-based approach (similar to segment routing) to time sensitive networking.
It furthermore proposes combining segment routing with this approach to TSN
resulting in a unified approach to forwarding and scheduling.

Y(J)S

---

From: "Jakob Heitz (jheitz)" <[email-masked]>
Date: Fri, 26 Feb 2021 04:47:28 +0000

Are they rare in wireless transmission? I believe some wireless protocols have lower layers that retransmit to provide reliability, but I don't know much about them.
My point is if you are going to provide a guarantee of time for delivery, you need to consider all factors that influence that time.

Also, what happens if you can't meet the deadline? Do you drop the packet or deliver with best effort?

Regards,
Jakob.

From: Yaakov Stein <[email-masked]>
Sent: Thursday, February 25, 2021 8:40 PM
To: Jakob Heitz (jheitz) <[email-masked]>; Pascal Thubert (pthubert) <pthubert=[email-masked]>
Cc: [email-masked]; Tianran Zhou <[email-masked]>; [email-masked]; [email-masked]
Subject: RE: new draft on segment routing approach to TSN

Jakob

Y(J)S

From: Jakob Heitz (jheitz) <[email-masked]<mailto:[email-masked]>>
Sent: 26/02/2021 01:36
To: Pascal Thubert (pthubert) <pthubert=[email-masked]<mailto:pthubert=[email-masked]>>; Yaakov Stein <[email-masked]<mailto:[email-masked]>>
Cc: [email-masked]<mailto:[email-masked]>; Tianran Zhou <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: RE: new draft on segment routing approach to TSN

Regards,
Jakob.

Hello Yaakov

Please see below:

Pascal,

Very cool;

Neat

Cool

  1.

Same here!

Pascal

Y(J)S

Pascal

Hi Yaakov,

Cheers,
Tianran

All,

Y(J)S

---

From: Yaakov Stein <[email-masked]>
Date: Fri, 26 Feb 2021 04:55:54 +0000

Of course they are common in wireless.
Cellular HARQ works just like that – if the FEC fixes the problem then fine, if not then the information is retransmitted.
And HARQ not only causes timing problems when there are errors, the HARQ SAW (Stop and Wait) process adds delay to all information transmitted.

However, I was mostly thinking about wireline cases.
Pascal pointed to the work being done in RAW, which is where this question needs to be addressed.

Y(J)S

From: Jakob Heitz (jheitz) <[email-masked]>
Sent: 26/02/2021 06:47
To: Yaakov Stein <[email-masked]>; Pascal Thubert (pthubert) <pthubert=[email-masked]>
Cc: [email-masked]; Tianran Zhou <[email-masked]>; [email-masked]; [email-masked]
Subject: RE: new draft on segment routing approach to TSN

Regards,
Jakob.

From: Yaakov Stein <[email-masked]<mailto:[email-masked]>>
Sent: Thursday, February 25, 2021 8:40 PM
To: Jakob Heitz (jheitz) <[email-masked]<mailto:[email-masked]>>; Pascal Thubert (pthubert) <pthubert=[email-masked]<mailto:pthubert=[email-masked]>>
Cc: [email-masked]<mailto:[email-masked]>; Tianran Zhou <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: RE: new draft on segment routing approach to TSN

Jakob

Y(J)S

Regards,
Jakob.

Hello Yaakov

Please see below:

Pascal,

Very cool;

Neat

Cool

  1.

Same here!

Pascal

Y(J)S

Whereever Yaakov decides to place it I’ll be there supporting the work. The draft itself is incredibly well-written and information-rich.
Note that there’s also work in RAW that mentions SR operation DetNet related operations (draft-pthubert-raw-architecture<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Fdraft-pthubert-raw-architecture-05&data=04%7C01%7Cyaakov_s%40rad.com%7Cae4ef21c0e8e4e5225c008d8da11a28a%7Cf9047108cc2c4e4897a343fad1b3bf9d%7C1%7C0%7C637499116573631718%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=bUFOFDAzIlH604zrCQ%2FAH99zNsOxn3o0EdNHslrNYJI%3D&reserved=0>). RAW has vested interest in intelligent forwarding decision, that would be the trademark vs. DetNet. With this draft, the forwarding is not based on Qbv schedule but the forwarder has some latitude as long as it matches the hop deadline. So RAW may be a good place.
And then there’s draft-chen-detnet-sr-based-bounded-latency<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Fdraft-chen-detnet-sr-based-bounded-latency-01&data=04%7C01%7Cyaakov_s%40rad.com%7Cae4ef21c0e8e4e5225c008d8da11a28a%7Cf9047108cc2c4e4897a343fad1b3bf9d%7C1%7C0%7C637499116573641712%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=aXDDtLjsa0m%2BR5FvQtDbvScRdfU5sZu5GuQN6hQiabA%3D&reserved=0>. Ideally all these related items would progress in the same room.

Also a few notes on the draft itself:
- maybe use latency instead of delay; it would be nice to maybe define delay as something else, e.g., the delay representing the time the packet spends queued in one hop vs. the latency that is end to end?
- not sure the term green wave is well understood by the public here; the draft gives the impression that the TSN path is faster than the best effort and involves no queueing. For the most part that is untrue; the latency is bounded but for most flows it is longer than best effort. Best effort can be really fast with passthrough in an empty network. The problem is the long tail and possibly congestion loss. For TSN, there can be very special flows that will traverse the city with all the lights green, but usually there’ll be queuing. The difference is that the queueing latency is constant and the overall latency is withing bounds.
- Time triggered is not the only TSN operation. I wonder what the draft would become with asynchronous shaper in mind. We designed (and as I must announce, patented as US9602420<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatents.google.com%2Fpatent%2FUS9602420&data=04%7C01%7Cyaakov_s%40rad.com%7Cae4ef21c0e8e4e5225c008d8da11a28a%7Cf9047108cc2c4e4897a343fad1b3bf9d%7C1%7C0%7C637499116573641712%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=VPsGSaolDUvNqJL8HcLhnISLG1VqLjlHNjbMASywZCo%3D&reserved=0>) a system very similar to the one proposed in the draft, but that is designed to adapt QoS depending on whether the packet is early or late vs. its schedule, and not tagging the schedule in the since the latency is considered end to end not hop by hop. The use case is slightly different since we apply this without a global controller and a provable guarantees all flows will meet the deadline – so not really detnet-, but more like a best effort that all flows meet their deadline in a stochastic environment. If Yaakov is interested, we can contribute on that aspect.

Pascal

Hi Yaakov,

Cheers,
Tianran

All,

I would like to call your attention to a new ID https://www.ietf.org/archive/id/draft-stein-srtsn-00.txt<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ietf.org%2Farchive%2Fid%2Fdraft-stein-srtsn-00.txt&data=04%7C01%7Cyaakov_s%40rad.com%7Cae4ef21c0e8e4e5225c008d8da11a28a%7Cf9047108cc2c4e4897a343fad1b3bf9d%7C1%7C0%7C637499116573651707%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=s%2BcWaL%2BW%2FyTV3lhy%2BVDLhjugaNVc9ryd5l%2BwoUSg0Yw%3D&reserved=0>
which describes using a stack-based approach (similar to segment routing) to time sensitive networking.
It furthermore proposes combining segment routing with this approach to TSN
resulting in a unified approach to forwarding and scheduling.

Y(J)S

---

From: "Jakob Heitz (jheitz)" <[email-masked]>
Date: Fri, 26 Feb 2021 04:59:47 +0000

Suppose you're sending packets out an interface and all the queues are empty. You got nothing to send.
What are you going to do with the dead air? Send idle patterns?
No.
Retransmit your previous queues. Just in case something got lost the preemptive retransmission will fix it quicker.
Then put a link layer sequence number and ACK and NACK numbers so you can manage what you retransmit.
This will only work on point-to-point. Ethernet's not broadcast on the physical layer anymore, is it?

Regards,
Jakob.

Sent: Thursday, February 25, 2021 8:56 PM

Y(J)S

From: Jakob Heitz (jheitz) <[email-masked]<mailto:[email-masked]>>
Sent: 26/02/2021 06:47
To: Yaakov Stein <[email-masked]<mailto:[email-masked]>>; Pascal Thubert (pthubert) <pthubert=[email-masked]<mailto:pthubert=[email-masked]>>
Cc: [email-masked]<mailto:[email-masked]>; Tianran Zhou <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: RE: new draft on segment routing approach to TSN

Regards,
Jakob.

Jakob

Y(J)S

Regards,
Jakob.

Hello Yaakov

Please see below:

Pascal,

Very cool;

Neat

Cool

  1.

Same here!

Pascal

Y(J)S

Pascal

Hi Yaakov,

Cheers,
Tianran

All,

Y(J)S

---

From: Yaakov Stein <[email-masked]>
Date: Fri, 26 Feb 2021 05:09:03 +0000

Tianran,

For some reason I didn’t see your email and thus never responded to it. My apologies.

1.       I believe that I address this question at the beginning of my draft.

Using a single deadline per packet at is suboptimal at any particular switch.

For example, say two packets arrive at a switch, one with only 20 microseconds left on its deadline

and one with 50. The switch would naturally schedule the 10 microsecond one first.

But what the switch doesn’t know is that it is the last hop for the 10 microsecond one

and there are only 2 microseconds from it to destination,

while the 50 microsecond packet still has 45 microseconds of physical time ahead of it!

  1.  I gave a specific algorithm in the draft and pointed to Andrews and Zhang who give another one.
  2.  I don’t think a service needs to support EDF, but perhaps a switch does.

I haven’t seen EDF support since the ATM days, but am proposing that perhaps the time has come to re-evaluate it.

However, I am not advocating for EDF in this draft, just for the stack.

I mention a variant of EDF which I believe is better, and am working on another variant which is even better.

They can all be built into hardware, although admitted are more complex than a simple queue.

But they may be simpler than a set of queues with time schedules like Qbv

and are much much simpler than monstrosities like MEF 10.3 token bucket with cross-color/cross-CoS sharing and coupling.
Y(J)S

Hi Yaakov,

Cheers,
Tianran

All,

Y(J)S

---

From: Yaakov Stein <[email-masked]>
Date: Fri, 26 Feb 2021 05:37:45 +0000

My point regarding HARQ was that this kind of retransmission can be more efficiently handled
at layers lower than where you see packets.
4G/5G has actually two mechanisms precisely of the type you suggest at two different layers,
but the magic happens on the physical signals (HARQ) or the SDUs (PDCP retransmission)
and is completely transparent to the user packets.

True, high speed Ethernet links are no longer physical broadcast domains
and one could insert sequence numbers into the frames and perform local retransmission.
This has been done in the past (I personally have a patent on such a method, although it also gives an alternative to numbering).
But if you are already numbering it might be even better to go all the way and use RFER.
(Or maybe even combine ACK/NACK retransmission and replication.)

Y(J)S

Sent: 26/02/2021 07:00

Regards,
Jakob.

Y(J)S

Regards,
Jakob.

Jakob

Y(J)S

Regards,
Jakob.

Hello Yaakov

Please see below:

Pascal,

Very cool;

Neat

Cool

  1.

Same here!

Pascal

Y(J)S

Whereever Yaakov decides to place it I’ll be there supporting the work. The draft itself is incredibly well-written and information-rich.
Note that there’s also work in RAW that mentions SR operation DetNet related operations (draft-pthubert-raw-architecture<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Fdraft-pthubert-raw-architecture-05&data=04%7C01%7Cyaakov_s%40rad.com%7C8cab9eb4a9ac43df556c08d8da135b8f%7Cf9047108cc2c4e4897a343fad1b3bf9d%7C1%7C0%7C637499123990426014%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=dEgNn4IHGjj3%2FRvVIOXSqDhQyXauw0fY7kutXi4bEPo%3D&reserved=0>). RAW has vested interest in intelligent forwarding decision, that would be the trademark vs. DetNet. With this draft, the forwarding is not based on Qbv schedule but the forwarder has some latitude as long as it matches the hop deadline. So RAW may be a good place.
And then there’s draft-chen-detnet-sr-based-bounded-latency<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Fdraft-chen-detnet-sr-based-bounded-latency-01&data=04%7C01%7Cyaakov_s%40rad.com%7C8cab9eb4a9ac43df556c08d8da135b8f%7Cf9047108cc2c4e4897a343fad1b3bf9d%7C1%7C0%7C637499123990436004%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=jiJvfdhiMnZVKaKCniqPxHqjnfKLr%2FUdZg3l4HMOLK0%3D&reserved=0>. Ideally all these related items would progress in the same room.

Also a few notes on the draft itself:
- maybe use latency instead of delay; it would be nice to maybe define delay as something else, e.g., the delay representing the time the packet spends queued in one hop vs. the latency that is end to end?
- not sure the term green wave is well understood by the public here; the draft gives the impression that the TSN path is faster than the best effort and involves no queueing. For the most part that is untrue; the latency is bounded but for most flows it is longer than best effort. Best effort can be really fast with passthrough in an empty network. The problem is the long tail and possibly congestion loss. For TSN, there can be very special flows that will traverse the city with all the lights green, but usually there’ll be queuing. The difference is that the queueing latency is constant and the overall latency is withing bounds.
- Time triggered is not the only TSN operation. I wonder what the draft would become with asynchronous shaper in mind. We designed (and as I must announce, patented as US9602420<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatents.google.com%2Fpatent%2FUS9602420&data=04%7C01%7Cyaakov_s%40rad.com%7C8cab9eb4a9ac43df556c08d8da135b8f%7Cf9047108cc2c4e4897a343fad1b3bf9d%7C1%7C0%7C637499123990436004%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=mSu2typomengGBz2sHO1%2B4kNez43HfPCDZCdBJZ1Pdc%3D&reserved=0>) a system very similar to the one proposed in the draft, but that is designed to adapt QoS depending on whether the packet is early or late vs. its schedule, and not tagging the schedule in the since the latency is considered end to end not hop by hop. The use case is slightly different since we apply this without a global controller and a provable guarantees all flows will meet the deadline – so not really detnet-, but more like a best effort that all flows meet their deadline in a stochastic environment. If Yaakov is interested, we can contribute on that aspect.

Pascal

Hi Yaakov,

Cheers,
Tianran

All,

I would like to call your attention to a new ID https://www.ietf.org/archive/id/draft-stein-srtsn-00.txt<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ietf.org%2Farchive%2Fid%2Fdraft-stein-srtsn-00.txt&data=04%7C01%7Cyaakov_s%40rad.com%7C8cab9eb4a9ac43df556c08d8da135b8f%7Cf9047108cc2c4e4897a343fad1b3bf9d%7C1%7C0%7C637499123990445994%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=NlU9Pzi3fgSJB365ALIdu1xk%2BtJZC2DEoCGoweQY4HQ%3D&reserved=0>
which describes using a stack-based approach (similar to segment routing) to time sensitive networking.
It furthermore proposes combining segment routing with this approach to TSN
resulting in a unified approach to forwarding and scheduling.

Y(J)S

---

From: Yaakov Stein <[email-masked]>
Date: Fri, 26 Feb 2021 05:47:40 +0000

Jakob

I forgot to address your first sentence.

Retransmission without NACK is an interesting idea, but I am not sure it really helps.

If there is nothing to be transmitted then you could go into a loop and retransmit packets over and over,
but I assume that this usually won’t happen.
So assume that you have a short idle window – which packet are you going to retransmit?
Without knowing which packet was not received correctly you have no idea.

So let’s assume that you reserve a full 50% of your BW and retransmit every packet.
Some packets received twice will differ – which one is correct? You can’t know without a FEC.
And if you are using a FEC it is more efficient to use a strong one then retransmit the whole packet!

So, let’s assume that you don’t use a FEC (maybe you don’t want the hassle of implementing one in hardware).
Then you need to send each packet at least 3 times to be able to do a majority decision.
So, you have reduced your BW efficiency to 33%.

I’m not saying that this is never justified. Lots of messages are sent 3 times for redundancy.
Going back to cellular, triple redundancy is used for critical control messages (in addition to other error correction means)/
But only for them. It is really suboptimal for everything else.

Y(J)S

From: Yaakov Stein
Sent: 26/02/2021 07:38

Y(J)S

Regards,
Jakob.

Y(J)S

Regards,
Jakob.

Jakob

Y(J)S

Regards,
Jakob.

Hello Yaakov

Please see below:

Pascal,

Very cool;

Neat

Cool

  1.

Same here!

Pascal

Y(J)S

Pascal

Hi Yaakov,

Cheers,
Tianran

All,

Y(J)S

---

From: "Jakob Heitz (jheitz)" <[email-masked]>
Date: Fri, 26 Feb 2021 05:55:58 +0000

Lots of ways. Another:
Use a circular buffer. As you transmit a packet put it into the buffer.
As it is acknowledged, remove it.
Just keep transmitting the buffer circularly.
If one is NACKed, retransmit it immediately.
The receiver can keep track of received sequence numbers.
They should be in order. Ack the latest sequence whenever there is a chance.
If the receiver sees a hole in the received sequence, NACK it.

Regards,
Jakob.

Sent: Thursday, February 25, 2021 9:48 PM

Jakob

Y(J)S

From: Yaakov Stein
Sent: 26/02/2021 07:38

Y(J)S

Regards,
Jakob.

Y(J)S

Regards,
Jakob.

Jakob

Y(J)S

Regards,
Jakob.

Hello Yaakov

Please see below:

Pascal,

Very cool;

Neat

Cool

  1.

Same here!

Pascal

Y(J)S

Pascal

Hi Yaakov,

Cheers,
Tianran

All,

Y(J)S

---

From: Yaakov Stein <[email-masked]>
Date: Fri, 26 Feb 2021 06:06:55 +0000

And I forgot to mention the real way this is handled in sophisticated wireless systems.

There shouldn’t be any idle time!

If there are less bits to be transmitted then the modulation order is reduced
increasing the margin and reducing the probability of error!

Even for NRZ Ethernet transmitting random frames that happen to the next in a cyclic buffer
is in general not the best option.
Not only does it not guarantee retransmission of a bad packet (unless is really very little to transmit),
it wastes a lot of energy on transmitting packets that have already been received.
If there is so little to transmit you should be powering down your transmitter (802.3az).

Y(J)S

Sent: 26/02/2021 07:56

Lots of ways. Another:

Regards,
Jakob.

Jakob

Y(J)S

From: Yaakov Stein
Sent: 26/02/2021 07:38

Y(J)S

Regards,
Jakob.

Y(J)S

Regards,
Jakob.

Jakob

Y(J)S

Regards,
Jakob.

Hello Yaakov

Please see below:

Pascal,

Very cool;

Neat

Cool

  1.

Same here!

Pascal

Y(J)S

Whereever Yaakov decides to place it I’ll be there supporting the work. The draft itself is incredibly well-written and information-rich.
Note that there’s also work in RAW that mentions SR operation DetNet related operations (draft-pthubert-raw-architecture<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Fdraft-pthubert-raw-architecture-05&data=04%7C01%7Cyaakov_s%40rad.com%7C7c35c6cd3c294bfa0c3a08d8da1b359a%7Cf9047108cc2c4e4897a343fad1b3bf9d%7C1%7C0%7C637499157699129811%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=SdSb0fY0Sz9L303hegarZn8vsWQTHl4btRH8OH8TvIs%3D&reserved=0>). RAW has vested interest in intelligent forwarding decision, that would be the trademark vs. DetNet. With this draft, the forwarding is not based on Qbv schedule but the forwarder has some latitude as long as it matches the hop deadline. So RAW may be a good place.
And then there’s draft-chen-detnet-sr-based-bounded-latency<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Fdraft-chen-detnet-sr-based-bounded-latency-01&data=04%7C01%7Cyaakov_s%40rad.com%7C7c35c6cd3c294bfa0c3a08d8da1b359a%7Cf9047108cc2c4e4897a343fad1b3bf9d%7C1%7C0%7C637499157699139798%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=VH2cFU4arYdrqyJ0THSOl2CSmdqorFRrOk0aSc%2FL6vg%3D&reserved=0>. Ideally all these related items would progress in the same room.

Also a few notes on the draft itself:
- maybe use latency instead of delay; it would be nice to maybe define delay as something else, e.g., the delay representing the time the packet spends queued in one hop vs. the latency that is end to end?
- not sure the term green wave is well understood by the public here; the draft gives the impression that the TSN path is faster than the best effort and involves no queueing. For the most part that is untrue; the latency is bounded but for most flows it is longer than best effort. Best effort can be really fast with passthrough in an empty network. The problem is the long tail and possibly congestion loss. For TSN, there can be very special flows that will traverse the city with all the lights green, but usually there’ll be queuing. The difference is that the queueing latency is constant and the overall latency is withing bounds.
- Time triggered is not the only TSN operation. I wonder what the draft would become with asynchronous shaper in mind. We designed (and as I must announce, patented as US9602420<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatents.google.com%2Fpatent%2FUS9602420&data=04%7C01%7Cyaakov_s%40rad.com%7C7c35c6cd3c294bfa0c3a08d8da1b359a%7Cf9047108cc2c4e4897a343fad1b3bf9d%7C1%7C0%7C637499157699149797%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=6JZDeqWc3chpJRaH5PTTdKWZvyeuDOFVhhGx%2BVA3Mw0%3D&reserved=0>) a system very similar to the one proposed in the draft, but that is designed to adapt QoS depending on whether the packet is early or late vs. its schedule, and not tagging the schedule in the since the latency is considered end to end not hop by hop. The use case is slightly different since we apply this without a global controller and a provable guarantees all flows will meet the deadline – so not really detnet-, but more like a best effort that all flows meet their deadline in a stochastic environment. If Yaakov is interested, we can contribute on that aspect.

Pascal

Hi Yaakov,

Cheers,
Tianran

All,

I would like to call your attention to a new ID https://www.ietf.org/archive/id/draft-stein-srtsn-00.txt<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ietf.org%2Farchive%2Fid%2Fdraft-stein-srtsn-00.txt&data=04%7C01%7Cyaakov_s%40rad.com%7C7c35c6cd3c294bfa0c3a08d8da1b359a%7Cf9047108cc2c4e4897a343fad1b3bf9d%7C1%7C0%7C637499157699149797%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=mE2eo7Ic3Iv%2BYuiFIgl2o32vgV0nLMkQedGLv%2BOk2n0%3D&reserved=0>
which describes using a stack-based approach (similar to segment routing) to time sensitive networking.
It furthermore proposes combining segment routing with this approach to TSN
resulting in a unified approach to forwarding and scheduling.

Y(J)S

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Fri, 26 Feb 2021 07:08:44 +0000

Hello Yaakov

Short term could you ask the RAW chairs for 15mn at the next IETF?

Wherever your work finally progresses that is a good start...

Take care,

Pascal

Le 26 févr. 2021 à 05:32, Yaakov Stein <[email-masked]> a écrit :

Pascal,

Y(J)S

Hello Yaakov

Please see below:

Pascal,

Very cool;

Neat

Cool

  1.

Same here!

Pascal

Y(J)S

Pascal

Hi Yaakov,

Cheers,
Tianran

All,

Y(J)S

---

From: "Jakob Heitz (jheitz)" <[email-masked]>
Date: Fri, 26 Feb 2021 07:24:26 +0000

i'm talking about point to point

Regards,
Jakob.

On Feb 25, 2021, at 10:08 PM, Yaakov Stein <[email-masked]> wrote:

Y(J)S

Lots of ways. Another:

Regards,
Jakob.

Jakob

Y(J)S

From: Yaakov Stein
Sent: 26/02/2021 07:38

Y(J)S

Regards,
Jakob.

Y(J)S

Regards,
Jakob.

Jakob

Y(J)S

Regards,
Jakob.

Hello Yaakov

Please see below:

Pascal,

Very cool;

Neat

Cool

  1.

Same here!

Pascal

Y(J)S

Pascal

Hi Yaakov,

Cheers,
Tianran

All,

Y(J)S

---

From: Yaakov Stein <[email-masked]>
Date: Fri, 26 Feb 2021 08:05:22 +0000

So was I.

Y(J)S

From: Jakob Heitz (jheitz) <[email-masked]>
Sent: 26/02/2021 09:24
To: Yaakov Stein <[email-masked]>
Cc: Pascal Thubert (pthubert) <pthubert=[email-masked]>; [email-masked]; Tianran Zhou <[email-masked]>; [email-masked]; [email-masked]
Subject: Re: new draft on segment routing approach to TSN

On Feb 25, 2021, at 10:08 PM, Yaakov Stein <[email-masked]<mailto:[email-masked]>> wrote:

Y(J)S

Lots of ways. Another:

Regards,
Jakob.

Jakob

Y(J)S

From: Yaakov Stein
Sent: 26/02/2021 07:38

Y(J)S

Regards,
Jakob.

Y(J)S

Regards,
Jakob.

Jakob

Y(J)S

Regards,
Jakob.

Hello Yaakov

Please see below:

Pascal,

Very cool;

Neat

Cool

  1.

Same here!

Pascal

Y(J)S

Whereever Yaakov decides to place it I’ll be there supporting the work. The draft itself is incredibly well-written and information-rich.
Note that there’s also work in RAW that mentions SR operation DetNet related operations (draft-pthubert-raw-architecture<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Fdraft-pthubert-raw-architecture-05&data=04%7C01%7Cyaakov_s%40rad.com%7C51b98cb39f0d45c099ca08d8da278fd3%7Cf9047108cc2c4e4897a343fad1b3bf9d%7C1%7C0%7C637499210756735148%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=Z4SoRo8LWfQCn5vP1k%2FEfsA%2BXhVTR%2BMUx6VkKIJzU%2B4%3D&reserved=0>). RAW has vested interest in intelligent forwarding decision, that would be the trademark vs. DetNet. With this draft, the forwarding is not based on Qbv schedule but the forwarder has some latitude as long as it matches the hop deadline. So RAW may be a good place.
And then there’s draft-chen-detnet-sr-based-bounded-latency<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Fdraft-chen-detnet-sr-based-bounded-latency-01&data=04%7C01%7Cyaakov_s%40rad.com%7C51b98cb39f0d45c099ca08d8da278fd3%7Cf9047108cc2c4e4897a343fad1b3bf9d%7C1%7C0%7C637499210756745135%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=h%2BhqzPfNogXRWUleHqkGo7ouAapuh1Sh93IOvci6wuA%3D&reserved=0>. Ideally all these related items would progress in the same room.

Also a few notes on the draft itself:
- maybe use latency instead of delay; it would be nice to maybe define delay as something else, e.g., the delay representing the time the packet spends queued in one hop vs. the latency that is end to end?
- not sure the term green wave is well understood by the public here; the draft gives the impression that the TSN path is faster than the best effort and involves no queueing. For the most part that is untrue; the latency is bounded but for most flows it is longer than best effort. Best effort can be really fast with passthrough in an empty network. The problem is the long tail and possibly congestion loss. For TSN, there can be very special flows that will traverse the city with all the lights green, but usually there’ll be queuing. The difference is that the queueing latency is constant and the overall latency is withing bounds.
- Time triggered is not the only TSN operation. I wonder what the draft would become with asynchronous shaper in mind. We designed (and as I must announce, patented as US9602420<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatents.google.com%2Fpatent%2FUS9602420&data=04%7C01%7Cyaakov_s%40rad.com%7C51b98cb39f0d45c099ca08d8da278fd3%7Cf9047108cc2c4e4897a343fad1b3bf9d%7C1%7C0%7C637499210756745135%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=hhaGeS%2FkTCDY7X2otyLrj3BTDRWRA1vR8Mw73CezMfc%3D&reserved=0>) a system very similar to the one proposed in the draft, but that is designed to adapt QoS depending on whether the packet is early or late vs. its schedule, and not tagging the schedule in the since the latency is considered end to end not hop by hop. The use case is slightly different since we apply this without a global controller and a provable guarantees all flows will meet the deadline – so not really detnet-, but more like a best effort that all flows meet their deadline in a stochastic environment. If Yaakov is interested, we can contribute on that aspect.

Pascal

Hi Yaakov,

Cheers,
Tianran

All,

I would like to call your attention to a new ID https://www.ietf.org/archive/id/draft-stein-srtsn-00.txt<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ietf.org%2Farchive%2Fid%2Fdraft-stein-srtsn-00.txt&data=04%7C01%7Cyaakov_s%40rad.com%7C51b98cb39f0d45c099ca08d8da278fd3%7Cf9047108cc2c4e4897a343fad1b3bf9d%7C1%7C0%7C637499210756755131%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=yOBPzTyNhM3qSrxcnr724wTYMjFvb0nduTpQO%2B5G4lM%3D&reserved=0>
which describes using a stack-based approach (similar to segment routing) to time sensitive networking.
It furthermore proposes combining segment routing with this approach to TSN
resulting in a unified approach to forwarding and scheduling.

Y(J)S

---

From: Tianran Zhou <[email-masked]>
Date: Fri, 26 Feb 2021 08:20:29 +0000

Hi Yaakov,

Thanks for your clarification.
Again I think this is an interesting work.
Please see inline.

Cheers,
Tianran

From: Yaakov Stein [mailto:[email-masked]]
Sent: Friday, February 26, 2021 1:09 PM
To: Tianran Zhou <[email-masked]>
Cc: [email-masked]; [email-masked]; [email-masked]
Subject: RE: new draft on segment routing approach to TSN

Tianran,

ZTR> Yes, you provided a valid use case in some sense.

But I have to say, IMO, the real-time system cares more about whether the deadline could meet.

So in practice, the task set is fixed, and all the latencies are calculated and guaranteed to meet the dead line. I do not care whether the packet could arrive earlier or not.

So, an deterministic and easy schedule make the evaluation/plan easier. And It’s better to see all the tasks could be accommodated in the plan.

ZTR> I can provide an use case, say a task could meet the 20 end to end deadline within 2 hops, one hop for 14, and the other for 6.

If we give split hop by hop deadlines, 10 for each.  That means the first hop with 14 delay cannot meet the 10 local deadline.

So for the first schedule, this task could be accommodated. But for the second one, this task cannot be accommodated.

2.       I gave a specific algorithm in the draft and pointed to Andrews and Zhang who give another one.

ZTR> Thanks for the pointer, I would like to look into them.

3.       I don’t think a service needs to support EDF, but perhaps a switch does.

ZTR> Not sure if my thoughts are common sense or not. ☺

IMO, the merit of Qbv, or TAS, is to isolate hard real-time, soft real-time, best effort tasks. So that they can accommodate in one switch.

This is one critical requirement for IT/OT integration.

EDF does not conflict with TAS. I worked on some hierarchical scheduling mechanisms many years ago, EDF is used in the time slot for hard real-time tasks.

Y(J)S

Hi Yaakov,

Cheers,
Tianran

All,

Y(J)S

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Fri, 26 Feb 2021 08:39:54 +0000

Hello Yaakov

If it was me I’d split the draft to take the deeper explanation of the why in a framework informational piece and leave the minimum in the standard track document.

Think that the implementer is a person of the art, and that the IESG will challenge every word in the standard track specification...

I d welcome text that explains that the PSE has to look at not only where to forward but also when to forward, in the RAW architecture/ framework draft.

And yes I often use train station and bus lines and connection to explain deterministic...

Take care,

Pascal

Pascal,

Y(J)S

Hello Yaakov

Please see below:

Pascal,

Very cool;

Neat

Cool

  1.

Same here!

Pascal

Y(J)S

Pascal

Hi Yaakov,

Cheers,
Tianran

All,

Y(J)S

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Fri, 26 Feb 2021 09:12:48 +0000

Hello Jakob

There’s a lot about that in the RAW architecture; please start there and then visit the DetNet architecture as well if you need to dig deeper.

 The bottom lines are:

- there will be multiple copies of a packet using all sorts of diversity including path diversity;
- the packets may be fragmented/ network coded to reduce the influence of very short term bit error events;
- there is no ARQ as typically thought of, though there can be more than a single TXOP scheduled for that frame over that wireless hop
- the tail end will typically receive more then one copy of a packet, think 2.x in average so as to almost never completely miss a packet.

Take care,

Pascal

Le 26 févr. 2021 à 06:00, Jakob Heitz (jheitz) <jheitz=[email-masked]> a écrit :

Regards,
Jakob.

Y(J)S

Regards,
Jakob.

Jakob

Y(J)S

Regards,
Jakob.

Hello Yaakov

Please see below:

Pascal,

Very cool;

Neat

Cool

  1.

Same here!

Pascal

Y(J)S

Pascal

Hi Yaakov,

Cheers,
Tianran

All,

Y(J)S

---

From: Yaakov Stein <[email-masked]>
Date: Fri, 26 Feb 2021 10:13:30 +0000

Tianran,

All valid points.

I do not doubt that there are RT cases with drop-dead deadlines,
but even they must live with rare losses (which is what a failure to meet deadline becomes).

As I said earlier the stack mechanism can replicate time gating behavior.
But time gating loses bandwidth efficiency and is really hard to scale up.

EDF in general will beat calendaring (i.e., has a lower expected latency),
but has a longer tail (which will be the aforementioned losses).
Given some statistics of packet transmission times one can calculate the percentage of packets
in that tail for a simple-to-setup approach with EDF.
It might be the case that some loss is better than not being able to calculate the schedule
and having to devote a lambda to each flow.

Y(J)S

From: Tianran Zhou <[email-masked]>
Sent: 26/02/2021 10:20
To: Yaakov Stein <[email-masked]>
Cc: [email-masked]; [email-masked]; [email-masked]
Subject: RE: new draft on segment routing approach to TSN

Cheers,
Tianran

From: Yaakov Stein [mailto:[email-masked]]
Sent: Friday, February 26, 2021 1:09 PM
To: Tianran Zhou <[email-masked]<mailto:[email-masked]>>
Cc: [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: RE: new draft on segment routing approach to TSN

Tianran,

For some reason I didn't see your email and thus never responded to it. My apologies.

But what the switch doesn't know is that it is the last hop for the 10 microsecond one

So, an deterministic and easy schedule make the evaluation/plan easier. And It's better to see all the tasks could be accommodated in the plan.

  1.  I don't think a service needs to support EDF, but perhaps a switch does.

I haven't seen EDF support since the ATM days, but am proposing that perhaps the time has come to re-evaluate it.

ZTR> Not sure if my thoughts are common sense or not. :)

Y(J)S

Hi Yaakov,

Cheers,
Tianran

All,

I would like to call your attention to a new ID https://www.ietf.org/archive/id/draft-stein-srtsn-00.txt<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ietf.org%2Farchive%2Fid%2Fdraft-stein-srtsn-00.txt&data=04%7C01%7Cyaakov_s%40rad.com%7Cad8cd800d9504f145a9a08d8da2f6322%7Cf9047108cc2c4e4897a343fad1b3bf9d%7C1%7C0%7C637499244365725326%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=xBIZ5ReTMzcAfx1TafhEl5wBlIzK6M8KskAQG%2BdtSZY%3D&reserved=0>
which describes using a stack-based approach (similar to segment routing) to time sensitive networking.
It furthermore proposes combining segment routing with this approach to TSN
resulting in a unified approach to forwarding and scheduling.

Y(J)S

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Fri, 26 Feb 2021 10:58:12 +0000

Your RFC should be able to signal drop dead but I believe that it would not be the decision of an intermediate hop. Missing a deadline between b and c may be recoverable between c and d...

Regards,

Pascal

Le 26 févr. 2021 à 11:14, Yaakov Stein <[email-masked]> a écrit :

Tianran,

All valid points.

Y(J)S

Cheers,
Tianran

Tianran,

  1.  I don’t think a service needs to support EDF, but perhaps a switch does.

Y(J)S

Hi Yaakov,

Cheers,
Tianran

All,

Y(J)S

_______________________________________________
detnet mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/detnet

---

From: Tianran Zhou <[email-masked]>
Date: Fri, 26 Feb 2021 11:55:24 +0000

Yes, Pascal, that is exactly the case I mentioned.

Best,
Tianran

________________________________

Sent from WeLink
发件人： Pascal Thubert (pthubert)<[email-masked]<mailto:[email-masked]>>
收件人： Yaakov Stein<[email-masked]<mailto:[email-masked]>>
抄送： Tianran Zhou<[email-masked]<mailto:[email-masked]>>;spring<[email-masked]<mailto:[email-masked]>>;detnet<[email-masked]<mailto:[email-masked]>>;pce<[email-masked]<mailto:[email-masked]>>
主题： Re: [Detnet] new draft on segment routing approach to TSN
时间： 2021-02-26 18:58:40

Regards,

Pascal

Tianran,

All valid points.

Y(J)S

Hi Yaakov,

Cheers,
Tianran

Tianran,

Y(J)S

Hi Yaakov,

Cheers,
Tianran

All,

Y(J)S

---

From: Balázs Varga A <[email-masked]>
Date: Fri, 26 Feb 2021 14:03:05 +0000

Hi Yaakov,

many thanks for this well-written and information-rich draft.
I have enjoyed to read it (and also the discussions on the lists). :--)))

Conceptual (clarification) questions:
1, What is the node behavior if deadtime expires? Drop? That requires a quite good design/allocation of the latency budget. Otherwise it may result in unnecessary loss of a packet being relative late at a hop, however that might be compensated by the remaining hops. (E.g., see your Figure 2, getting 35 microsecond at R1 instead of the designed 25, but R2,R3,R4 adds only 10-10-10 extra microseconds). You may consider to add a "late flag" to the packet instead of a drop ...

2, Do I understand it correctly that the described solution is practically a new per-hop-behavior? Does it have no tools to "influence" the arrival time at the next hops? In end-to-end latency design the DetNet/TSN functions are often used to "adapt" the arrival of packets belonging to a DetNet flow/TSN stream at given network points. If "latency design" requires such a "time shift" then the described method has to be combined with other DetNet/TSN tools.

3, Design of a guaranteed upper bound usually requires deterministic arrival at each node along the path. In this solution the flow becomes less-an-less deterministic as it reaches the destination. I mean e.g., as per Figure 2, arrival time range at R1 is 2-2 usec, at R2 is 26-51 usec, at R3 is 70-120 usec and at R4 is 92-167 usec. So, the flow is getting less-and-less deterministic at each hop. That can make deterministic design of other flows passing R4 more difficult or even impossible in some scenarios. Have I missed something here?

4, Wire-speed timestamping and calculation: I would be interested in your view regarding how realistic are (1) the wire speed timestamping capability and (2) adding multiple calculated deadline to packets. Both are required for this solution. DetNet flows can have high bandwidth, not like 1588 packets.

Specific comments (Section 2-3):
5, Time gated Queuing [802.1Qbv] allows multiple gates to be open simultaneously. Furthermore preemption [802.1Qbu] can be combined with Qbv. So I cannot fully agree with your evaluation of time-aware scheduling. (Section 2, "However, time-aware scheduling suffers from two major disadvantages.") Of course with a bad design one may shoot in his/her own leg, but in well-designed scenarios efficiency loss can be eliminated without expensive computation.

6, Deadtime calculation is also complex if impact of other DetNet flows must be considered. The pre-calculation of individual "local" deadlines may need to be re-calculated when flows added to the network and they are using some common links. So I cannot agree with your statement. (Section 3, "... Since the ingress router inserts the deadline stack into the packet headers, no other router needs to be aware of the requirements of the time sensitive flows.  Hence admitting a new flow only requires updating the information base of the ingress router."). Adding a flow may result in updating many ingress routers' configuration.

Some topics for further discussions/considerations:
7, Impact of some DetNet functions (e.g., PREOF [RFC8655]) needs further discussions. The usage of replication/elimination impacts the deadline calculation. Disjoint paths used by member flows requires different label stack and deadlines. Furthermore, these path specific labels+deadlines must be added by the replication node (PRF). So at least, a combination of PRF with B-SID and computing/adding deadlines (ingress SRTSN function) seems to be necessary in your solution.

8, The ever-green topic ( and not the green-wave :--))) ): label stack size. You need multiple labels per hop and the label stack contains them for each hop. Could result in a quite big label stack to be pushed at the ingress (nx10s of labels).

I am happy to have further discussions on this interesting idea

Thanks & Cheers
Bala'zs

From: detnet <[email-masked]> On Behalf Of Yaakov Stein
Sent: Tuesday, February 23, 2021 2:14 PM
To: [email-masked]; [email-masked]; [email-masked]
Subject: [Detnet] new draft on segment routing approach to TSN

All,

Y(J)S

---

From: Toerless Eckert <[email-masked]>
Date: Sat, 27 Feb 2021 19:19:08 +0100

To add to Pascals reading list, check out:

https://www.ietf.org/archive/id/draft-stein-srtsn-00.txt

This isn't meaning to endorse all the opinions and conclusions offered, but while probably
not being complete, i found it to be is AFAIK the most comprehensive survey for large scale
network bounded latency.

---

From: Toerless Eckert <[email-masked]>
Date: Sat, 27 Feb 2021 22:12:16 +0100

whow... that was a mayor coyp&paste blunder...
No idea how that went through...
apologies! Correct URL:

https://arxiv.org/pdf/1905.08478.pdf

-- 
---
[email-masked]

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Sat, 27 Feb 2021 21:52:06 +0000

Many thanks Toerless!

Pascal

> Le 27 févr. 2021 à 22:12, Toerless Eckert <[email-masked]> a écrit :
>