Subject: Re: [Detnet] Policing/shaping and related calculus [was draft-varga-detnet-pof (sorry, lot of input)]

From: Toerless Eckert <[email-masked]>
Date: Sat, 23 Oct 2021 16:36:10 +0200

Your mail was replied on 15th September:                                                                             
https://mailarchive.ietf.org/arch/msg/detnet/wjotZ71P7uhEeoMpbRwQQdy-FE0/                                            

I am talking to a reply for my email from the 21st, whcih was a reply to your email frm the 15h .. see appended.

Maybe better to move this to a chat on gather.town or the like....

Cheers
    Toerless

On Tue, Sep 21, 2021 at 07:53:10PM +0200, Toerless Eckert wrote:
> On Wed, Sep 15, 2021 at 02:19:14PM +0000, Balázs Varga A wrote:
> > Hi Toerless,
> > 
> > I have renamed the thread as it is about policing/shaping and not PEF/POF.
> > 
> > I disagree. Whether there is a burst after PEF/POF depends on several factors, like
> > topology, path delay difference, flow characteristics (bulk vs. intermittent flow), etc.
> 
> The calculus for every known deterministic latency regulation system
> only support well-known envelops for flows. So lets assume we use
> UBS/TSN-ATS to manage latency, then we have bitrate+burstsize admitted
> flows (plus max packet size, which we can ignore for examples).
> Most simple case AFAIK.
> 
> For any given such calculus, i think we can determine the worst case
> latency added by PEF independent of topology for the UBS calculus.
> 
> I am not sure we can determine a tighter added latency bound for
> general topologies beyond that. There may be specific topologies where
> tighter bounds could be calculated, but i think thats a tertiary option
> (aka: not very interesting).
> 
> > Again, POF/PEF may cause burst at the DetNet service sub-layer, but DetNet forwarding 
> > sub-layer provides the functions to cope with those bursts.
> 
> How ? PEF/POF operate on the DetNet service sub-layer. The DetNet
> forwarding layer just contributes to the burstyness/latency issue
> that PEF/POF have to deal with.
> 
> > I think we should be systematic and first focus on the root cause of the "jitter" You refer to.
> 
> Of course.
> 
> > Namely, packets of a flow takes different path over the network. Under normal conditions
> > packets over the shortest path wins in PREOF functions. If a network situation (e.g., failure)
> > causes that packets of a flow take different path, then extra-jitter occurs. This may happen in 
> > non-DetNet networks as well. 
> 
> Triple redundancy = 3 copies, three paths (just to have more fun than 2).
> Path 1 has lowest latency, path 2 second longest, path 3 longest.
> Maximum burst size of flow is 1 packet (of some size). Aka: flow is just a regular
> interval sequence of packets of same size.
> 
> Packet 1 is lost on path 1,2, packet 2 is lost on path 2.
> 
> I can easily have the case where the frequency of the packets for
> the flow and the path latency difference is such that packet
> will arrive in order but at pretty much exactly the same time,
> so that PEF will release packet 1,2,3 at exactly the same time.
> 
> Eg: packets repeat every 100 usec, non-queuing path latency is
> path 2 takes 100 usec longer than path 1, and path 3 is 200
> usec longer than path 1.
> 
> Now assume that we have PEF be followed by standard TSN-ATS
> regulator and priority queue. The flow presented after PEF to
> those two stages wold not be compliant to the CIR/PIR of the flow,
> and hence we would see buffer overrun in either stage and/or
> unexpected high latency in them.
> 
> I think this burst accumulation is also higher over
> the flow agregate if it was just 2 copies, but i have
> not managed to find a simple explained example. Even
> with just a single path, i can create the rare case of
> two packets of such a simple example flow to be immediately
> following each other after the prior hop FIFO stage,
> hitting the regulator, but with PEF they would not
> even be separated in time by the serialization of the
> one link, but could arrive fully in parallel on the
> following hop.
> 
> > Policing and shaping of a flow may target these scenarios. Policing and shaping functions
> > belong to the forwarding sub-layer in DetNet context.
> 
> This is a side discussion, but IMHO important:
>   Where in RFC8655 is it actually defined at which layer policing
>   and shaping belong to. I can not read that from the RFC. Rather,
>   4.5 for example reads to me like the opposite: managing latency is
>   done at DetNet layer. IMHO, managing latency/buffers must happen
>   at both layers depending on the path setup, but it would be good
>   if we do have agreemnt and maybe even consider making some RFC8655bis or
>   other TBD RFC stronger on this spec aspect.
> 
> If we went with what you said, and my example, we would have
> a DetNet flow of 1 packet burst size up to the PER point, and
> after PER we would have a DetNet flow with a burst size of
> 3 packets. I don't think this is what we want. And shaping out
> a PER burst accumulation in the forwarding sub-layer also sounds
> like inappropriate layer conflation.
> 
> IMHO, what we need is for example instead of the normal
> sized regulator for UBS a regulator that shapes the flow
> again to it defined rate, but that has more buffer. Probably
> 3 times the buffer size as would be required without the 3-path
> diversity. The normal UBS regulator effectively does not incur
> added latency over the easily calculated per-hop latency of
> the prior stage FIFO, but this regulator given how it would
> shape out three times the permissible burst size would
> incur additional latency. And i don't think anyone has done
> the calculus for that (professional pessimist here: either i
> am right or someone has, which would be great). I would venture
> to guess the added latency is 2 times the flows burst-size
> in its bitrate.
> 
> > Latency bound calculation always take the worst case scenario. The bounded latency draft 
> > covers regulators and calculates with them. See e.g., Figure-1, Figure-2 and calculation of
> > "end_to_end_latency_bound_of_flow_f".
> 
> The draft only mentions PREOF once. I don't think it considers
> the PEF impact onto the latency calculation model at all.
> 
> > Have I missed something?
> 
> See above.
> 
> Cheers
>     Toerless
> > 
> > Cheers
> > Bala'zs
> > 
> > 
> > -----Original Message-----
> > From: Toerless Eckert <[email-masked]> 
> > Sent: Wednesday, September 15, 2021 12:23 AM
> > To: Balázs Varga A <[email-masked]>
> > Cc: [email-masked]
> > Subject: Re: [Detnet] draft-varga-detnet-pof (sorry, lot of input)
> > 
> > If i do PEF from e.g. 3 copies of a flow, i can easily create a burst size that is 3 times of the permitted burst size of the flow on the PEF router. This is higher than you would ever get without PEF. None of the calculus in our bounded latency draft covers this.
> > 
> > If you where to simply put PEF in front of an unchanged GuaranteedService
> > (GS) shaper stage or TSN-ATS/UBS regulator+priority-queuing stage, then you would require a lot more buffer and create a lot more latency than what the standard calculus for these mechanisms calculates.
> > 
> > So. how do you think DetNet model covers this burstyness and latency of PEF ?
> > 
> > My gut feeling is that there is no lower-bounded-latency solution than to insert a PEF specific regulator after PEF that re-shapes the flow to its reserved envelope. But i think the actual calculus for PEF latency is TBD.
> > 
> > Cheers
> >     Toerless
> > 
> > On Fri, Sep 10, 2021 at 02:08:31PM +0000, Balázs Varga A wrote:
> > > Hi Toerless,
> > > 
> > > Yes, POF/PEF may cause burst at the DetNet service sub-layer, but 
> > > DetNet forwarding sub-layer provides the functions to cope with those 
> > > bursts.
> > > 
> > > Adding additional buffering to PEF/POF would be a duplication of 
> > > functions and increase latency.
> > > 
> > > Please note that TSN/DetNet consider always the worst case scenario. 
> > > That means practically, resource allocation based on PCR.
> > > 
> > > Cheers
> > > Bala'zs
> > > 
> > > -----Original Message-----
> > > From: Toerless Eckert <[email-masked]>
> > > Sent: Wednesday, September 8, 2021 9:36 PM
> > > To: Balázs Varga A <[email-masked]>
> > > Cc: [email-masked]
> > > Subject: Re: [Detnet] draft-varga-detnet-pof (sorry, lot of input)
> > > 
> > > Being a bit delayed by the draft i was promising, i just wanted to bring up in the context of your work one point i stumbled across:
> > > 
> > > Lets say we have:
> > > 
> > >       T-PE1 -> S-PE1 -> LSR-P -> S-PE2 -> T-PE2
> > > 
> > > Lets assume the detnet domain runs from T-PE1 to T-PE2 But now (and i think/hop this is covered by the DetNet architecture), i want to do Packet Duplication Function on S-PE1 and Packet Elimination Function of S-PE2.
> > > 
> > > When i do introduce Packet Ordering Function in conjunction with Packet Elimination on S-PE2, i think that this could create bursts of packets that can exceed the traffic contract known to the owner of the flow (e.g.: CIR/PIR etc..), because this burstyness depends for example on the differential latency of the for example two paths/copies sent between S-PE1 and S-PE2.
> > > 
> > > In effect, i think that one can not simply rely on the bounded latency iand buffering model of any pre-existing boundd latency algorithm as long as that only takes the flow CIR/PIR etc into account. 
> > > 
> > > Aka: In your POF model, where PEF is first performed, those bursts go into the POF, and if we want to hide all this POF/PEF impact from some bounded-latency operation happening after the POF, i think we need to include into the POF some more buffering functionality that reduces burstyness to what the egres bounded latency function can hanle without becoming yet another more complex ingres-edge "gate"
> > > function.
> > > 
> > > Cheers
> > >     Toerless
> > > 
> > > On Wed, Aug 11, 2021 at 11:49:24PM +0200, Toerless Eckert wrote:
> > > > High level answer:
> > > > 
> > > > If we would start to describe the management interface, e.g.:
> > > > what we want be able to configure and observe about all of PREOF 
> > > > (maybe starting with POF), then i think we will immediately get to 
> > > > the point that we expect a much more explicit description of what 
> > > > PREOF/POF has to do so that it can comply with that management 
> > > > inerface.
> > > > 
> > > > I am saying this because when i did design features in the past it 
> > > > did work out best by first starting to think about how the operator 
> > > > should see the feaure. Otherwise a lot was forgotten. In the IETF i 
> > > > did admittedly wiggle myself around doing this 100% correctly, by 
> > > > just informally referring to the CLI/config aspects (such as 
> > > > RFC8994), but that was primarily because for that work the manaement 
> > > > work is really big, so it didn't first. Writing a yang module into a 
> > > > POF draft itself should be rather painless in comparison.
> > > > 
> > > > From there on out, we can start to see how well we could describe 
> > > > the per-hop-behavior, if that is how we are allowed to call it.
> > > > 
> > > > Details hat may be missing are about stuff like: what if a packet 
> > > > arrives whose sequence number does not fall into the window of 
> > > > sequence numbers that the POF currently accepts ? Ok, we discard it.
> > > > We update a counter tracking these events. Do we update our POF 
> > > > state machinery ? Maybe that needs to be configurable.
> > > > Should the window-size of sequence number be configurable, etc. pp
> > > > 
> > > > Good info about TSN. Thanks
> > > > 
> > > > Toerless
> > > > 
> > > > 
> > > > On Wed, Aug 04, 2021 at 10:15:15AM +0000, Balázs Varga A wrote:
> > > > > Hi Toerless,
> > > > > 
> > > > > Many thanks for the comments. Some clarifications/reactions:
> > > > > 
> > > > > a) "per hop behavior"
> > > > > In general I agree, "unexpected behavior of DetNet products" must be avoided.
> > > > > We have already POF specifics in DetNet RFCs:
> > > > > - RFC8655: introduces the term POF
> > > > > - RFC8938: gives some characteristics of POF (uses sequence 
> > > > > numbers, range of packet order protection, buffer resources)
> > > > > - RFC8964: defines POF packet processing (section 4.2.2.3) What is 
> > > > > missing in your view?
> > > > > 
> > > > > Regarding terminology I also dislike "input vs. outbut timing for 
> > > > > traffic " as POF is not about explicit timing but order of packets.
> > > > > 
> > > > > b) Management interface
> > > > > Yes, management is an important topic. If I understand your 
> > > > > comment correctly You are more concerned about the 
> > > > > operation/maintenance related counters/parameters. I think it is a 
> > > > > valid comment, but in my view it might be better to discuss it in the OAM related drafts.
> > > > > In this draft all the POF algorithm's specific parameters are 
> > > > > defined. E.g., required buffer space can be derived from these parameters and the network/flow specifics.
> > > > > 
> > > > > c) TSN standards about ordering function There were discussions 
> > > > > about ordering in TSN, but nothing standardized.
> > > > > 
> > > > > d) "Another algorithm option: configure for every packet copy path a fixed time delay"
> > > > > I think this topic needs more clarification. Some comments:
> > > > > - I do not see differences in topologies used by TSN and by DetNet. 
> > > > > Rings are also typical in TSN networks.
> > > > > - We have a path specific delay parameter defined in section "4.4. 
> > > > > The Advanced POF Algorithm", namely "POFMaxDelay_i".
> > > > > - I am not sure your proposal is a POF function or a PDF (Packet Delay Function).
> > > > > 
> > > > > Again, thanks for the review and the comments.
> > > > > 
> > > > > Cheers
> > > > > Bala'zs
> > > > > 
> > > > > -----Original Message-----
> > > > > From: detnet <[email-masked]> On Behalf Of Toerless 
> > > > > Eckert
> > > > > Sent: Saturday, July 31, 2021 12:41 AM
> > > > > To: [email-masked]
> > > > > Subject: [Detnet] draft-varga-detnet-pof (sorry, lot of input)
> > > > > 
> > > > > Balazs, *:
> > > > > 
> > > > > I very much support work on POF, however:
> > > > > 
> > > > > I think we should aim higher than informational guidelines
> > > > > 
> > > > > I think we can build a normative document on two interfaces:
> > > > > 
> > > > > a) "per hop behavior"
> > > > > 
> > > > >    We can stanardize the externally observable behavior
> > > > >    between arriving and sent packets on the node.
> > > > > 
> > > > >    This was regularily done for DiffServ Traffic Classes in
> > > > >    TSVWG, where it is calle Per Hop Behavior. When i tried
> > > > >    to apply this term to the input vs. outbut timing for
> > > > >    traffic from what today is an IntServ flow (DetNet),
> > > > >    David Black was somewhat concerned about re-use of the
> > > > >    term, which is why i wrote "per hop behavior". IMHO,
> > > > >    it is perfectly valid to also call this PHB, but i will
> > > > >    leave it up to David to recommend appropriate terminology.
> > > > > 
> > > > >    I am only interested to be able to get as little as possible
> > > > >    unexpecte behavior of DetNet products, and we can only 
> > > > >    arrive that if we standardize the behavior.
> > > > > 
> > > > > b) Management interface
> > > > > 
> > > > >    Yang module specifying what the operator 
> > > > >    can observe about the function about the POF function, e.g: 
> > > > > 
> > > > >    "static" parameters such as how many buffers there are available,
> > > > >      algorithm (you do have algorithm in the draft)
> > > > >     - maybe read/write if configurable.
> > > > > 
> > > > >    "dynamic parameters" - read-only for monitoring, troubleshooting:
> > > > >      - buffers dynamically used,
> > > > >      - latency between first and further copies
> > > > >      - missing packets
> > > > >      - ...
> > > > > 
> > > > > Now wrt to other content:
> > > > > 
> > > > > c) It would be lovely to have a section summarizing what if anythng
> > > > >    TSN did standardize about their ordering function for all those
> > > > >    IETF participants too lazy to read all that documentation
> > > > >    and/or rejecting the notion of having to pay money to read a document.
> > > > > 
> > > > >    (informational, appendix).
> > > > > 
> > > > > d) I think it would be great to have anoher algorithm option
> > > > >    by which one could configure for every packet copy path
> > > > >    (assuming we can identify that in the forwardin plane)
> > > > >    a fixed time delay.
> > > > >   
> > > > >    The reason for this, and why i think it was not an issue in
> > > > >    TSN (with its typical topologies) is that in DetNets
> > > > >    we much more likely will have rings with significant path 
> > > > >    latency differences between A and B path,
> > > > >    and whenever the A (shorter) path looses a packet, we would wait
> > > > >    until the B path packet arrives, resulting in a lot of
> > > > >    jitter. If the DetNet controller could configure 
> > > > >    the B-A pah lateny difference as a delay, we could get rid
> > > > >    of this problem.
> > > > > 
> > > > > Still have to dig deeper into the algo details you proposed for more comments.
> > > > > 
> > > > > Cheers
> > > > >     Toerless
> > > > > 
> > > > > _______________________________________________
> > > > > detnet mailing list
> > > > > [email-masked]
> > > > > https://www.ietf.org/mailman/listinfo/detnet
> > > > 
> > > > --
> > > > ---
> > > > [email-masked]
> > > 
> > > --
> > > ---
> > > [email-masked]
> > 
> > --
> > ---
> > [email-masked]
> 
> -- 
> ---
> [email-masked]
> 
> _______________________________________________
> detnet mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/detnet

-- 
---
[email-masked]