Subject: [Detnet] updating the flow information model draft

From: János Farkas <[email-masked]>
Date: Wed, 4 Jul 2018 20:12:03 +0200

Dear all,

I think the flow information model draft needs some updates to bring it 
in-line with the recent developments in the WG. I’d like to have WG 
discussion on the list and at IETF 102 before making any updates.

1, Current version of draft-ietf-detnet-flow-information-model-01

Section 5.2 Service Parameters includes in order delivery, which is a 
binary value in draft-ietf-detnet-flow-information-model-01 depending on 
whether or not in order delivery is required for the given service. This 
requires refinements based on the developments we had in 
draft-ietf-detnet-architecture-06.

The refinement would involve introducing a numerical service parameter, 
which could be called “MisorderingTolerance” or alike to replace the 
current binary parameter.

2, Recent updates of draft-ietf-detnet-architecture-06

draft-ietf-detnet-architecture-06 provides a finer grade DetNet QoS 
parameter by introducing :

Section 3.1:
“o An upper bound on out-of-order packet delivery. It is worth
noting that some DetNet applications are unable to tolerate any
out-of-order delivery.”

Section 3.2.2.1. In-Order Delivery:
“Out-of-order packet delivery can be a side effect of service
protection. Packets delivered out-of-order impact the amount of
buffering needed at the destination to properly process the received
data. Such packets also influence the jitter of a flow. The DetNet
service includes maximum allowed misordering as a constraint. Zero
misordering would be a valid service constraint to reflect that the
end system(s) of the flow cannot tolerate any out-of-order delivery.
Service protection may provide a mechanism to support in-order
delivery.”

The architecture document does not provide further details on purpose. 
It is the job of the flow information model draft to go into the details.

3, “MisorderingTolerance” related thoughts

The DetNet QoS requirements are correlated to each other. For instance, 
a packet received with a too big delay or jitter is practically a lost 
packet for the application. As for misordering, it is related, e.g., to 
jitter as explained in the architecture document. (Well, misordering 
above the tolerable limit can be considered loss.) The correlations can 
be taken into account when defining the QoS requirements for a DetNet 
service, and also when providing the given DetNet service by the DetNet 
network.

Actually, misordering and jitter can be decreased with a similar tool, 
e.g., a playout buffer. It is a bit more complex for reordering and 
requires sequencing information carried in the packets.

I see two approaches to express maximum allowed misordering:
a) number of misordered packets
b) time

Both of them have their pros and cons.

I propose to follow a) for multiple reasons.

Maximum allowed/tolerable misordering could be defined along:
“maximum number of misordered packets, e.g., a delta of sequence numbers 
between the highest sequence number that was just received and the 
lowest sequence that is acceptable”
(credit goes to Pascal for initial wording!)

This is a relatively simple definition and also simple to measure, 
detect, and verify. Furthermore, timing requirement is already defined 
as part of the jitter requirement, no need to duplicate. In addition, 
timing may not be as important as the order of packets for some 
applications.

What do you think?

Regards,
Janos

---

From: "qiangli (D)" <[email-masked]>
Date: Thu, 12 Jul 2018 06:39:43 +0000

Hi Janos,

I agree with you that the information model draft needs to add some "in order delivery" related content. And I personally prefer your first suggestion, using the number of misordered packets to express the misodering. 

Besides, I have two other questions:

--what is the relationship between "section 5.2 service parameters" and "section 10.3 user to network requirements", aren't they both on the interface between user and network? One interface two usages, depend on which direction you look at it. From network to user is service, in the opposite direction is requirement, but either way, they will have the same parameters.

-- Do we need to expand information model draft for jitter? I couldn't find the description of jitter in the current version. However according to the PS draft, jitter is one of the important features of detnet flow, see below:

"....along the multi-hop path with such characteristics as low latency and ultra-low jitter, reordering and/or replication and elimination of packets.....", 
as well as "Need absolute guarantees of minimum and maximum latency end-to-end across the network; sometimes a tight jitter is required as well;"

Best regards,

Cristina QIANG

-----Original Message-----
From: detnet [mailto:[email-masked]] On Behalf Of János Farkas
Sent: Thursday, July 05, 2018 2:12 AM
To: DetNet WG <[email-masked]>
Subject: [Detnet] updating the flow information model draft

Dear all,

I think the flow information model draft needs some updates to bring it in-line with the recent developments in the WG. I’d like to have WG discussion on the list and at IETF 102 before making any updates.

Section 5.2 Service Parameters includes in order delivery, which is a binary value in draft-ietf-detnet-flow-information-model-01 depending on whether or not in order delivery is required for the given service. This requires refinements based on the developments we had in draft-ietf-detnet-architecture-06.

The refinement would involve introducing a numerical service parameter, which could be called “MisorderingTolerance” or alike to replace the current binary parameter.

draft-ietf-detnet-architecture-06 provides a finer grade DetNet QoS parameter by introducing :

Section 3.1:
“o An upper bound on out-of-order packet delivery. It is worth noting that some DetNet applications are unable to tolerate any out-of-order delivery.”

Section 3.2.2.1. In-Order Delivery:
“Out-of-order packet delivery can be a side effect of service protection. Packets delivered out-of-order impact the amount of buffering needed at the destination to properly process the received data. Such packets also influence the jitter of a flow. The DetNet service includes maximum allowed misordering as a constraint. Zero misordering would be a valid service constraint to reflect that the end system(s) of the flow cannot tolerate any out-of-order delivery.
Service protection may provide a mechanism to support in-order delivery.”

The DetNet QoS requirements are correlated to each other. For instance, a packet received with a too big delay or jitter is practically a lost packet for the application. As for misordering, it is related, e.g., to jitter as explained in the architecture document. (Well, misordering above the tolerable limit can be considered loss.) The correlations can be taken into account when defining the QoS requirements for a DetNet service, and also when providing the given DetNet service by the DetNet network.

Actually, misordering and jitter can be decreased with a similar tool, e.g., a playout buffer. It is a bit more complex for reordering and requires sequencing information carried in the packets.

Maximum allowed/tolerable misordering could be defined along:
“maximum number of misordered packets, e.g., a delta of sequence numbers between the highest sequence number that was just received and the lowest sequence that is acceptable”
(credit goes to Pascal for initial wording!)

This is a relatively simple definition and also simple to measure, detect, and verify. Furthermore, timing requirement is already defined as part of the jitter requirement, no need to duplicate. In addition, timing may not be as important as the order of packets for some applications.

What do you think?

Regards,
Janos

_______________________________________________
detnet mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/detnet

---

From: Balázs Varga A <[email-masked]>
Date: Fri, 13 Jul 2018 12:39:59 +0000

Hi Cristina,

Regarding the questions:
- chapter 5.2 vs 10.3:
Right. We have to consolidate and harmonize text in those chapters. We have to consider multiple sets of parameters.
(1) User's view: parameters that are defined by the user for its flow (it may be application specific), 
(2) Network's view: service parameters the network(/operator) provides for a DetNet flow
(3) User network interaction: parameters the user signals to the network. 
These sets and the relationship between them have to be sorted out in order to have a consolidated text.

- jitter:
Agree, we have to include it. So far in the discussions the main focus was bounded delay. That may imply somewhat jitter, but I agree we need more explicit focus on jitter.

Cheers
Bala'zs

-----Original Message-----
From: detnet <[email-masked]> On Behalf Of qiangli (D)
Sent: 2018. július 12. 8:40
To: Janos Farkas <[email-masked]>; DetNet WG <[email-masked]>
Subject: Re: [Detnet] updating the flow information model draft

Hi Janos,

"....along the multi-hop path with such characteristics as low latency and ultra-low jitter, reordering and/or replication and elimination of packets.....", as well as "Need absolute guarantees of minimum and maximum latency end-to-end across the network; sometimes a tight jitter is required as well;"

Best regards,

Cristina QIANG

-----Original Message-----
From: detnet [mailto:[email-masked]] On Behalf Of János Farkas
Sent: Thursday, July 05, 2018 2:12 AM
To: DetNet WG <[email-masked]>
Subject: [Detnet] updating the flow information model draft

Dear all,

Section 3.1:

What do you think?

Regards,
Janos

---

From: "qiangli (D)" <[email-masked]>
Date: Mon, 16 Jul 2018 03:45:41 +0000

Hi Bala'zs,

It's a good idea to classify parameters and reorganize text as these three categories. In User network interaction set, besides what you said I think maybe the parameters the network to users also need to be taken in to account, such as some monitoring, OAM results, etc.

Best regards,

Cristina QIANG

-----Original Message-----
From: Balázs Varga A [mailto:[email-masked]] 
Sent: Friday, July 13, 2018 8:40 PM
To: qiangli (D) <[email-masked]>; Janos Farkas <[email-masked]>; DetNet WG <[email-masked]>
Subject: RE: [Detnet] updating the flow information model draft

Hi Cristina,

Regarding the questions:
- chapter 5.2 vs 10.3:

- jitter:

Cheers
Bala'zs

-----Original Message-----
From: detnet <[email-masked]> On Behalf Of qiangli (D)
Sent: 2018. július 12. 8:40
To: Janos Farkas <[email-masked]>; DetNet WG <[email-masked]>
Subject: Re: [Detnet] updating the flow information model draft

Hi Janos,

Best regards,

Cristina QIANG

-----Original Message-----
From: detnet [mailto:[email-masked]] On Behalf Of János Farkas
Sent: Thursday, July 05, 2018 2:12 AM
To: DetNet WG <[email-masked]>
Subject: [Detnet] updating the flow information model draft

Dear all,

Section 3.1:

What do you think?

Regards,
Janos

---

From: Rodney Cummings <[email-masked]>
Date: Tue, 17 Jul 2018 14:56:25 +0000

Hi Janos,

I don't necessarily agree with the proposals to add in-order delivery and/or probability-of-packet-loss to the DetNet information model(s).

We need to keep the user in mind. The user-side of the information model must provide parameters that:
1. Are known by a user without any knowledge of the network details.
2. Can be practically provided by the network independent of the application/user data protocol.

Item 2 means that we cannot assume a specific protocol on top of IP and/or layer-2 packets. Maybe it is UDP. Maybe it is TCP. Maybe it is layer-2 (e.g. IEEE Std 1722). We all know that TCP has a lower probability-of-packet-loss than UDP, but that is not relevant to the DetNet information model.

In that context, if the information model provides a probability-of-packet-loss parameter, how do we propose that the user will compute that number? I would argue that the user does not have sufficient knowledge to do this. What's more... how would the network compute the number that is delivered by the network? For example, if Ethernet cabling is installed in a noisy environment, probability-of-packet-loss is probably higher, but I don't see how a central controller would be aware of it.

To be practical, we need to provide parameters that are associated with things the user actually knows. For example, the user knows whether he/she can deployed IEEE Std 802.1CB technology in switches/routers, and the user know how many redundant paths have been deployed. Therefore, if we have a flow parameter like number-of-redundant-paths, that is something that the user can understand, and it is something that the network can check for and configure.

As for in-order delivery, I haven't seen a proposal for how that would be delivered by the network. If/until we have such a proposal, adding it now would do nothing but provide a false promise. I would argue that we should wait until a standard for in-order delivery exists, and then provide this parameter.

Rodney

> Sent: Thursday, July 12, 2018 1:40 AM
> To: János Farkas <[email-masked]>; DetNet WG <[email-masked]>
> I agree with you that the information model draft needs to add some "in
> order delivery" related content. And I personally prefer your first
> suggestion, using the number of misordered packets to express the
> misodering.
> 
> --what is the relationship between "section 5.2 service parameters" and
> "section 10.3 user to network requirements", aren't they both on the
> interface between user and network? One interface two usages, depend on
> which direction you look at it. From network to user is service, in the
> opposite direction is requirement, but either way, they will have the same
> parameters.
> 
> -- Do we need to expand information model draft for jitter? I couldn't
> find the description of jitter in the current version. However according
> to the PS draft, jitter is one of the important features of detnet flow,
> see below:
> 
> "....along the multi-hop path with such characteristics as low latency and
> ultra-low jitter, reordering and/or replication and elimination of
> packets.....",
> as well as "Need absolute guarantees of minimum and maximum latency end-
> to-end across the network; sometimes a tight jitter is required as well;"
> 
> 
> Best regards,
> 
> Cristina QIANG
> 
> requires refinements based on the developments we had in draft-ietf-
> detnet-architecture-06.
> 
> “o An upper bound on out-of-order packet delivery. It is worth noting that
> some DetNet applications are unable to tolerate any out-of-order
> delivery.”
> 
> “Out-of-order packet delivery can be a side effect of service protection.
> Packets delivered out-of-order impact the amount of buffering needed at
> the destination to properly process the received data. Such packets also
> influence the jitter of a flow. The DetNet service includes maximum
> allowed misordering as a constraint. Zero misordering would be a valid
> service constraint to reflect that the end system(s) of the flow cannot
> tolerate any out-of-order delivery.
> The DetNet QoS requirements are correlated to each other. For instance, a
> packet received with a too big delay or jitter is practically a lost
> jitter as explained in the architecture document. (Well, misordering above
> the tolerable limit can be considered loss.) The correlations can be taken
> into account when defining the QoS requirements for a DetNet service, and
> also when providing the given DetNet service by the DetNet network.
> 
> between the highest sequence number that was just received and the lowest
> sequence that is acceptable”
> This is a relatively simple definition and also simple to measure, detect,
> and verify. Furthermore, timing requirement is already defined as part of
> the jitter requirement, no need to duplicate. In addition, timing may not
> be as important as the order of packets for some applications.
> 
> What do you think?
> 
> Regards,
> Janos
> 
> https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__www.ietf.org_mailman_listinfo_detnet&d=DwIGaQ&c=I_0YwoKy7z5LMTVdyO6YCi
> E2uzI1jjZZuIPelcSjixA&r=WA71sf2o7Dw7CbYhFt24DPjt3lJuupswWYdnboKbZ8k&m=LfnP
> f6Uf2mcshZCd4Hhh86TbHNNdk94xnvtrMY80gXk&s=tsOfM8UGcBXWlUPiGg7XegkpgP5foZGt
> ulwtpGszb_g&e=
> https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__www.ietf.org_mailman_listinfo_detnet&d=DwIGaQ&c=I_0YwoKy7z5LMTVdyO6YCi
> E2uzI1jjZZuIPelcSjixA&r=WA71sf2o7Dw7CbYhFt24DPjt3lJuupswWYdnboKbZ8k&m=LfnP
> f6Uf2mcshZCd4Hhh86TbHNNdk94xnvtrMY80gXk&s=tsOfM8UGcBXWlUPiGg7XegkpgP5foZGt
> ulwtpGszb_g&e=

---

From: Norman Finn <[email-masked]>
Date: Tue, 17 Jul 2018 20:33:53 +0000

I have a problem with the idea of the user expressing a requirement for worst-case packet misordering.  My problem is that I'm not sure what the network can do with that information.

Either you are using PR/PE or not.  If you are NOT using PR/PE, then you are presumably using some routing/bridging protocol to heal the network after a failure.  The out-of-order problem will occur when, some time after a failure caused a loss of packets and the use of a long path A-->B, healing the failure and thus opening a shorter path causes early deliveries and hence out-of-order packets.  In order to predict the worst-case scenario for out-of-order delivery, so that the network can pass or fail the reservation request, one would have to determine the worst-case packets in flight for the longest path A-->B after multiple failures in the network, and then look at the fastest possible healing to the shortest possible path to get the worst-case out-of-order difference.  It might be possible to calculate this in a single-ring topology, but in a more complex topology, I have my doubts that it can be performed reliably.  Without a reliable way to calculate the worst case, the user's misordering requirements are irrelevant.

If you ARE using PR/PE, then either you are using a bandwidth reservation or you are not.  If you are NOT using a bandwidth reservation, then all bets are off.  It is difficult to predict the worst-case difference in packets-in-flight for the two paths, and hence the worst-case sequence delta.  With nothing to compare the user's misording requirements to, the requirements are irrelevant.

If you ARE using PR/PE and you ARE using bandwidth reservations, then as you are making the reservations, you can determine the worst-case time differential on the different paths before a merge point, you know the maximum data rates, and you can determine the buffer space needed to put the packets in order.    If the merge system has insufficient buffer space to accomplish the merge, then the user's misordering requirements are irrelevant.  That buffer space is required in order to both a) prevent overloading the bandwidth reservation of the merged stream and b) not drop packets due to congestion of the merge port, whether or not in-order delivery is desired.    If the merge buffer space is available, then there is no problem with full in-order delivery, and the user's misordering requirements are irrelevant.

Finally, what do we imagine the network to do other than refuse the reservation in the very few cases where this is possible?  If it discards packets with larger differences, then you are doing nothing that the receiver cannot do for itself.  You also are taking on the problem of knowing when to resume transmission, detecting when the sequence number is reset, etc.  These are not trivial problems, and wind up needing help from the sequencing source.  This is better left to the application to sort out.

To sum up, I think that user requirements are much better expressed, and the network's ability to conform to those requirements are better measured, in terms of best- and worst-case latency and latency variation.

-- Norm
________________________________________
From: detnet [[email-masked]] on behalf of János Farkas [[email-masked]]
Sent: Wednesday, July 04, 2018 11:12 AM
To: DetNet WG
Subject: [Detnet] updating the flow information model draft

Dear all,

Section 3.1:

What do you think?

Regards,
Janos

---

From: Lou Berger <[email-masked]>
Date: Tue, 17 Jul 2018 16:39:25 -0400

Norm,

     In the IP case, worst-case packet misordering can also impact 
multi-path forwarding behavior.  An implementation (IP or MPLS data 
planes) can always satisfy a non-zero reordering requirement with paths 
that don't reorder.

Lou

---

From: Norman Finn <[email-masked]>
Date: Tue, 17 Jul 2018 23:01:40 +0000

Binary "reordering is/not acceptable" I can see some use for.  It's the numbers that I don't know how to deal with.

-- Norm

________________________________________
From: Lou Berger [[email-masked]]
Sent: Tuesday, July 17, 2018 1:39 PM
To: Norman Finn; János Farkas; DetNet WG
Subject: Re: [Detnet] updating the flow information model draft

Norm,

Lou

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Tue, 17 Jul 2018 23:30:04 +0000

It’s tough...

I can see that a bounced number of bytes received in disorder can be fixed by a recomposition buffer facing the application 

I can see that a bounded time disorder at the rate of the flow translates in the above bytes.

Sadly none of the above is easy to deal with in the network where we see frames and sequence numbers.

A wasteful approximation is to divide the application buffer by mtu and then we get the number of frames we are talking about.

This does not seem to bright but then if the transport is smart enough to pack the data in bunches close to mtu, as much as possible within an acceptable time budget, then we can define an easy SLA both for ingress filtering and for reordering...

Regards,

Pascal

> Le 17 juil. 2018 à 19:02, Norman Finn <[email-masked]> a écrit :
> 
>> From: detnet [[email-masked]] on behalf of János Farkas [[email-masked]]

---

From: "qiangli (D)" <[email-masked]>
Date: Wed, 18 Jul 2018 02:44:48 +0000

Hi,

The root cause of different opinions I can see here is the mixing of information model and service model.

I tried to redraw Figure 1 of information model draft as follows, messy display in txt format. 
---Between user and network management/control entity is service model, here we assume user has limited knowledge of detnet network, hence user can only say "I want to purchase a detnet service for my application, please give me "very low" out-of-order transmission". 
---Then insides the management/control entity, "very low" will be translated into more technical language for example "less than 0.1% packet disordering" according to the information model. Next, management/control entity should do another work -- select a specific technology that can satisfy this requirement, and generates configuration parameters. 
---At last, these parameters will be set to devices/network nodes through the device configuration model.

 User                     Network Operator
                       +--------------------------------+
  /\   service model    | information  model  |   management/control entity
 /  \<--------------------->   |                    |  
 ------  [I want a detnet  | [ "very low"-> less than|  
       service with "very | 0.1% packet          |
       low" out-of-order] | disordering]         |
                       +-------------------------------+
                                      |
                                      |
                                      |
                                      |   device configuration
                                      |   model
                       +----------------------+-------------------------
                       |              |               |
                       |              |               |
                       |              |               |
                       |              |               |
                       |              |               |
                       v              v               v
              Network+---+          +---+           +---+
                     |   |          |   |           |   |
              Nodes  +---+          +---+           +---+

Best regards,

Cristina QIANG

-----Original Message-----
From: Rodney Cummings [mailto:[email-masked]] 
Sent: Tuesday, July 17, 2018 10:56 PM
To: qiangli (D) <[email-masked]>; János Farkas <[email-masked]>; DetNet WG <[email-masked]>
Subject: RE: [Detnet] updating the flow information model draft

Hi Janos,

Rodney

> I agree with you that the information model draft needs to add some 
> "in order delivery" related content. And I personally prefer your 
> first suggestion, using the number of misordered packets to express 
> the misodering.
> 
> --what is the relationship between "section 5.2 service parameters" 
> and "section 10.3 user to network requirements", aren't they both on 
> the interface between user and network? One interface two usages, 
> depend on which direction you look at it. From network to user is 
> service, in the opposite direction is requirement, but either way, 
> they will have the same parameters.
> 
> find the description of jitter in the current version. However 
> according to the PS draft, jitter is one of the important features of 
> detnet flow, see below:
> 
> "....along the multi-hop path with such characteristics as low latency 
> and ultra-low jitter, reordering and/or replication and elimination of 
> packets.....", as well as "Need absolute guarantees of minimum and 
> maximum latency end- to-end across the network; sometimes a tight 
> jitter is required as well;"
> 
> 
> Best regards,
> 
> Cristina QIANG
> 
> I think the flow information model draft needs some updates to bring 
> it in-line with the recent developments in the WG. I’d like to have WG 
> binary value in draft-ietf-detnet-flow-information-model-01 depending 
> on whether or not in order delivery is required for the given service. 
> This requires refinements based on the developments we had in 
> draft-ietf- detnet-architecture-06.
> 
> The refinement would involve introducing a numerical service 
> parameter, which could be called “MisorderingTolerance” or alike to 
> replace the current binary parameter.
> 
> 
> “o An upper bound on out-of-order packet delivery. It is worth noting 
> that some DetNet applications are unable to tolerate any out-of-order 
> delivery.”
> 
> Packets delivered out-of-order impact the amount of buffering needed 
> at the destination to properly process the received data. Such packets 
> also influence the jitter of a flow. The DetNet service includes 
> maximum allowed misordering as a constraint. Zero misordering would be 
> a valid service constraint to reflect that the end system(s) of the 
> flow cannot tolerate any out-of-order delivery.
> The DetNet QoS requirements are correlated to each other. For 
> instance, a packet received with a too big delay or jitter is 
> practically a lost packet for the application. As for misordering, it 
> is related, e.g., to jitter as explained in the architecture document. 
> (Well, misordering above the tolerable limit can be considered loss.) 
> The correlations can be taken into account when defining the QoS 
> requirements for a DetNet service, and also when providing the given DetNet service by the DetNet network.
> 
> “maximum number of misordered packets, e.g., a delta of sequence 
> numbers between the highest sequence number that was just received and 
> the lowest sequence that is acceptable”
> timing may not be as important as the order of packets for some applications.
> 
> What do you think?
> 
> Regards,
> Janos
>

---

From: "qiangli (D)" <[email-masked]>
Date: Wed, 18 Jul 2018 03:25:47 +0000

Regarding to buffer, I have a question. If the low jitter could be achieved through low latency + buffer or not? Is there any real-time application that doesn't tolerate buffer but requires low jitter?

Best regards,

Cristina QIANG

-----Original Message-----
From: detnet [mailto:[email-masked]] On Behalf Of Pascal Thubert (pthubert)
Sent: Wednesday, July 18, 2018 7:30 AM
To: Norman Finn <[email-masked]>
Cc: DetNet WG <[email-masked]>; Lou Berger <[email-masked]>; János Farkas <[email-masked]>
Subject: Re: [Detnet] updating the flow information model draft

It’s tough...

Regards,

Pascal

> planes) can always satisfy a non-zero reordering requirement with 
> paths that don't reorder.
> 
> Lou
> 
>> it in-line with the recent developments in the WG. I’d like to have 
>> WG discussion on the list and at IETF 102 before making any updates.
>> 
>> on whether or not in order delivery is required for the given 
>> service. This requires refinements based on the developments we had 
>> in draft-ietf-detnet-architecture-06.
>> 
>> is related, e.g., to jitter as explained in the architecture 
>> document. (Well, misordering above the tolerable limit can be 
>> considered loss.) The correlations can be taken into account when 
>> defining the QoS requirements for a DetNet service, and also when 
>> providing the given DetNet service by the DetNet network.
>> 
>> Actually, misordering and jitter can be decreased with a similar 
>> tool, e.g., a playout buffer. It is a bit more complex for reordering 
>> and requires sequencing information carried in the packets.
>> 
>> numbers between the highest sequence number that was just received 
>> and the lowest sequence that is acceptable”
>> detect, and verify. Furthermore, timing requirement is already 
>> defined as part of the jitter requirement, no need to duplicate. In 
>> addition, timing may not be as important as the order of packets for 
>> some applications.
>> 
>> What do you think?
>> 
>> Regards,
>> Janos
>>

---

From: Toerless Eckert <[email-masked]>
Date: Wed, 18 Jul 2018 07:59:45 +0200

> Agree, we have to include it. So far in the discussions the main focus was bounded delay. That may imply somewhat jitter, but I agree we need more explicit focus on jitter.

What are the delay parameters ?

For deterministic service they should be min-delay...max-delay delay.
(max-min) is then the 'deterministic' jitter ?!

I guess one can also define a tighter stochastic jitter for some
percentile of traffic based on further detail understanding about the
stochastic nature of the jitter: For 100% of pckets jitter is min..max,
but for 95% its even better (less jitter). Just not sure whether
those additional metric are useful/necessary for detnet.

Cheers
    Toerless
> I think the flow information model draft needs some updates to bring it in-line with the recent developments in the WG. I???d like to have WG discussion on the list and at IETF 102 before making any updates.
> 
> The refinement would involve introducing a numerical service parameter, which could be called ???MisorderingTolerance??? or alike to replace the current binary parameter.
> 
> 
> ???o An upper bound on out-of-order packet delivery. It is worth noting that some DetNet applications are unable to tolerate any out-of-order delivery.???
> 
> ???Out-of-order packet delivery can be a side effect of service protection. Packets delivered out-of-order impact the amount of buffering needed at the destination to properly process the received data. Such packets also influence the jitter of a flow. The DetNet service includes maximum allowed misordering as a constraint. Zero misordering would be a valid service constraint to reflect that the end system(s) of the flow cannot tolerate any out-of-order delivery.
> Service protection may provide a mechanism to support in-order delivery.???
> 
> 3, ???MisorderingTolerance??? related thoughts
> 
> ???maximum number of misordered packets, e.g., a delta of sequence numbers between the highest sequence number that was just received and the lowest sequence that is acceptable???

-- 
---
[email-masked]

---

From: Norman Finn <[email-masked]>
Date: Wed, 18 Jul 2018 15:38:18 +0000

I agree with you, Toerless.

There are many ways to specify jitter, and some facilitate negotiation between talker/listener and network.

The listener can reasonably specify a worst case value of ((max latency) - (min latency)), because that is related to his receive buffer space requirements.

Either the talker or the listener may want to specify a max latency.

Obviously, specifying max and (max - min) is equivalent to specifying max and min, but if the actual latency reported by the network is better than the requested max, that probably doesn't change the (max - min) difference, so there may be some small advantage in the negotiation procedure to use max and (max - min) as the parameters.

Perhaps I am blinded by the truly real-time applications, like industrial, but I don't think that a stochastic jitter specification is useful for DetNet.  "DetNet" is supposed to be deterministic.  Packet loss due to cosmic rays in inevitably a matter of chance, but delivery time is not.

-- Norm

________________________________________
From: detnet [[email-masked]] on behalf of Toerless Eckert [[email-masked]]
Sent: Tuesday, July 17, 2018 10:59 PM
To: Bal?zs Varga A
Cc: DetNet WG; Janos Farkas; qiangli (D)
Subject: Re: [Detnet] updating the flow information model draft

Cheers
    Toerless

--
---
[email-masked]

---

From: Janos Farkas <[email-masked]>
Date: Wed, 18 Jul 2018 21:23:15 +0000

Hi Rodney,

I often start from the most complicated approach, i.e., when we really have and SLA over the UNI. I know that there are simpler cases; however, if we cover the most complicated case, then simpler ones fall out.

There is nothing new with respect to loss requirement. No change in the draft. The loss requirement as a concept is not new. The user on the user side of the UNI knows the loss rate its application can tolerate.

Knowing the maximum loss tolerance, it can be determined from the loss requirement what type of service protection is needed, e.g., whether or not replication and elimination is needed.

I think both 1) and 2) are valid with respect to loss requirement.

There has been further discussions on in-order delivery. I reply to that. I think both 1) and 2) are valid with respect to in order delivery as well, e.g., a playout buffer that reorders packets can be used by the network to implement the Packet Ordering Function (POF) of PREOF.

Regards,
Janos

-----Original Message-----
From: Rodney Cummings <[email-masked]> 
Sent: Tuesday, July 17, 2018 4:56 PM
To: qiangli (D) <[email-masked]>; Janos Farkas <[email-masked]>; DetNet WG <[email-masked]>
Subject: RE: [Detnet] updating the flow information model draft

Hi Janos,

Rodney

---

From: Janos Farkas <[email-masked]>
Date: Wed, 18 Jul 2018 21:23:19 +0000

Hi,

I have no hard feelings.

A couple of thoughts.

To some extent this is a comment on the architecture document, where maximum allowed misordering has been recently added. I believe the text in the architecture document is fine even if it is only a binary value in practice. It is worth double checking.

As I mentioned in my previous mail, we may look at it from SLA perspective.
SLA may include parameters for the DetNet network between two UNIs like:
- maximum latency
- maximum delay variation 
- max tolerable loss
- max tolerable misordering

These are all parameters to reflect what the application can tolerate. 
We are talking about well described flows, e.g., CBR.
If the max misordering is not infinite but another positive integer number N, it reflects that N packets being out of order are not a problem, e.g., the application has a buffer to store and reorder N packets.

I believe if there is requirement on maximum misordering (e.g., N), then restoration is off the table; explicit routes are needed with service protection that allows the dimensioning for reordering. It may fall out of the loss requirement that the service protection applied should be PREOF. It is possible to determine worst case misordering for PREOF (e.g., W) as Norm described. If N < W, then it is enough to have space for N packets in the reordering playout buffer instead of W. So it is useful information for the network. Otherwise, the playout buffer has to be W unless there is no misorderign requirement at all.

Misordering and delay variation are correlated; however, somewhat different. For instance the delay variation requirement can be looser than the delay variation requirement if the delay variation caused by reordering of N packets is below the delay variation requirements. For instance a flow may have zero misordering tolerance but still allow some packet delay variation.
 
Regards,
Janos

-----Original Message-----
From: Norman Finn <[email-masked]> 
Sent: Wednesday, July 18, 2018 1:02 AM
To: Lou Berger <[email-masked]>; Janos Farkas <[email-masked]>; DetNet WG <[email-masked]>
Subject: RE: [Detnet] updating the flow information model draft

-- Norm

Norm,

     In the IP case, worst-case packet misordering can also impact multi-path forwarding behavior.  An implementation (IP or MPLS data
planes) can always satisfy a non-zero reordering requirement with paths that don't reorder.

Lou

On 7/17/2018 4:33 PM, Norman Finn wrote:
> it in-line with the recent developments in the WG. I'd like to have WG 
> parameter, which could be called "MisorderingTolerance" or alike to 
> "o An upper bound on out-of-order packet delivery. It is worth noting 
> delivery."
> "Out-of-order packet delivery can be a side effect of service 
> delivery."
> 3, "MisorderingTolerance" related thoughts
>
> "maximum number of misordered packets, e.g., a delta of sequence 
> the lowest sequence that is acceptable"

---

From: Janos Farkas <[email-masked]>
Date: Wed, 18 Jul 2018 21:29:24 +0000

Yes, max and min delay are classic description of delay variation, I'd follow that.
Thanks,
Janos

-----Original Message-----
From: Norman Finn <[email-masked]> 
Sent: Wednesday, July 18, 2018 5:38 PM
To: Toerless Eckert <[email-masked]>; Balázs Varga A <[email-masked]>
Cc: DetNet WG <[email-masked]>; Janos Farkas <[email-masked]>; qiangli (D) <[email-masked]>
Subject: RE: [Detnet] updating the flow information model draft

-- Norm

I guess one can also define a tighter stochastic jitter for some percentile of traffic based on further detail understanding about the stochastic nature of the jitter: For 100% of pckets jitter is min..max, but for 95% its even better (less jitter). Just not sure whether those additional metric are useful/necessary for detnet.

Cheers
    Toerless

--
---
[email-masked]

---

From: Lou Berger <[email-masked]>
Date: Thu, 19 Jul 2018 10:27:39 -0400

> Binary "reordering is/not acceptable" I can see some use for.  It's the numbers that I don't know how to deal with.
There are different technology related/specific solutions here.  The 
fundamental point  is to allow for it in the architecture and modeling.

Lou

---

From: Norman Finn <[email-masked]>
Date: Fri, 20 Jul 2018 18:50:12 +0000

Lou:  Perhaps you could expand on your "different technologies" comment.

Short explanation:

The amount of misordering, and hence the resources (including choice of algorithm) required to re-order packets, is known the the network, not the listener.  The application already specifies a min/max latency in order to bound his receive buffer requirements.  I would favor 1) adding the ability of the network to report the worst-case misordering, 2) keeping the binary "application can/cannot tolerate misordering" parameter, 3) having a binary "network can/cannot ensure ordering".

Long explanation:

I stand by my arguments about 5/100 vs. no/yes.  If it's possible for the network to misorder packets, the resources required to reorder them, given the path and the bandwidth/packet size spec for the flow, is certainly known to the network, but probably not to the application.  Reporting the worst-case misordering to the application would be useful.  But, allowing the application to specify "5" when the network is likely to deliver 100 simply complicates the reservation process.  Of course, it's easy to compare the application's requirement to the network's number, but that is not the issue.  Having the parameter merely begs a pile of questions for the application:

  - What number do I specify to maximize my chances of success?
  - How many buffers might the network want me to have?
  - Does that mean I need to know the topology?  Line lengths?
  - How do I compare my need for receiver buffers to my other needs for memory space?
  - Which device, me or the edge node, would find it easier/cheaper to increase the number of reordering buffers?
  - Does the edge node even have reordering capability?
  - Will the network configuration produce misordering, at all?

My claim is that most of these questions are settled much more easily with the current yes/no requirement plus a report from the network on worst-case misordering, and we can see a path for negotiating/refusing/remaking the reservation.  I further claim that turning the yes/no into a number will complicate the process without bringing anything significant to the range of results available. 

For example, are we suggesting that the network could buffer just enough to reduce the 100 to 5?  I don't think it worthwhile to support the (I think infrequent) case where the listener's capabilities to handle some misordering would make a significant difference in the resources required for the network to reduce misordering to that level, but not eliminate it.

I would rather not add a "misordering limit" parameter.  It will inevitably generate requirements for the control protocols support it, and I claim there is no clear use for that information.  Yes/no parameters for network and application capabilities, and a report on the possible misordering number from the network, is sufficient.

-- Norm

________________________________________
From: detnet [[email-masked]] on behalf of Lou Berger [[email-masked]]
Sent: Thursday, July 19, 2018 7:27 AM
To: Norman Finn; János Farkas; DetNet WG
Subject: Re: [Detnet] updating the flow information model draft

Lou

---

From: "Black, David" <[email-masked]>
Date: Mon, 23 Jul 2018 15:51:11 +0000

Norm,

Let me check whether I understand the approach here.

It sounds like the goal is to push reassembly of out-of-order packets and any resulting delays to the endpoints, particularly the receiver.   It looks like the network responds to a path reservation request by indicating the degree of reordering that is possible on that path, from which the application can then figure out how much reordering buffering is needed and the impacts thereof.  In particular the network would not need to add buffering for reordering within the network.

Is that consistent with what's proposed below?

Thanks, --David

> From: detnet [mailto:[email-masked]] On Behalf Of Norman Finn
> Sent: Friday, July 20, 2018 2:50 PM
> To: Lou Berger; János Farkas; DetNet WG
> The amount of misordering, and hence the resources (including choice of
> algorithm) required to re-order packets, is known the the network, not the
> listener.  The application already specifies a min/max latency in order to
> bound his receive buffer requirements.  I would favor 1) adding the ability of
> the network to report the worst-case misordering, 2) keeping the binary
> "application can/cannot tolerate misordering" parameter, 3) having a binary
> "network can/cannot ensure ordering".
> 
> Long explanation:
> 
> I stand by my arguments about 5/100 vs. no/yes.  If it's possible for the
> network to misorder packets, the resources required to reorder them, given
> the path and the bandwidth/packet size spec for the flow, is certainly known
> to the network, but probably not to the application.  Reporting the worst-
> case misordering to the application would be useful.  But, allowing the
> application to specify "5" when the network is likely to deliver 100 simply
> complicates the reservation process.  Of course, it's easy to compare the
> application's requirement to the network's number, but that is not the issue.
> Having the parameter merely begs a pile of questions for the application:
> 
>   - Which device, me or the edge node, would find it easier/cheaper to
> increase the number of reordering buffers?
> My claim is that most of these questions are settled much more easily with
> the current yes/no requirement plus a report from the network on worst-
> case misordering, and we can see a path for negotiating/refusing/remaking
> the reservation.  I further claim that turning the yes/no into a number will
> complicate the process without bringing anything significant to the range of
> results available.
> 
> For example, are we suggesting that the network could buffer just enough to
> reduce the 100 to 5?  I don't think it worthwhile to support the (I think
> infrequent) case where the listener's capabilities to handle some misordering
> would make a significant difference in the resources required for the
> network to reduce misordering to that level, but not eliminate it.
> 
> I would rather not add a "misordering limit" parameter.  It will inevitably
> generate requirements for the control protocols support it, and I claim there
> is no clear use for that information.  Yes/no parameters for network and
> application capabilities, and a report on the possible misordering number
> from the network, is sufficient.
> 
> -- Norm
> 
> > Binary "reordering is/not acceptable" I can see some use for.  It's the
> numbers that I don't know how to deal with.
> >> I have a problem with the idea of the user expressing a requirement for
> worst-case packet misordering.  My problem is that I'm not sure what the
> network can do with that information.
> >>
> >> Either you are using PR/PE or not.  If you are NOT using PR/PE, then you
> are presumably using some routing/bridging protocol to heal the network
> after a failure.  The out-of-order problem will occur when, some time after a
> failure caused a loss of packets and the use of a long path A-->B, healing the
> failure and thus opening a shorter path causes early deliveries and hence
> out-of-order packets.  In order to predict the worst-case scenario for out-of-
> order delivery, so that the network can pass or fail the reservation request,
> one would have to determine the worst-case packets in flight for the longest
> path A-->B after multiple failures in the network, and then look at the fastest
> possible healing to the shortest possible path to get the worst-case out-of-
> order difference.  It might be possible to calculate this in a single-ring
> topology, but in a more complex topology, I have my doubts that it can be
> performed reliably.  Without a reliable way to calculate the worst case, the
> user's misordering requirements are irrelevant.
> >>
> >> If you ARE using PR/PE, then either you are using a bandwidth reservation
> or you are not.  If you are NOT using a bandwidth reservation, then all bets
> are off.  It is difficult to predict the worst-case difference in packets-in-flight
> for the two paths, and hence the worst-case sequence delta.  With nothing
> to compare the user's misording requirements to, the requirements are
> irrelevant.
> >>
> >> If you ARE using PR/PE and you ARE using bandwidth reservations, then as
> you are making the reservations, you can determine the worst-case time
> differential on the different paths before a merge point, you know the
> maximum data rates, and you can determine the buffer space needed to put
> the packets in order.    If the merge system has insufficient buffer space to
> accomplish the merge, then the user's misordering requirements are
> irrelevant.  That buffer space is required in order to both a) prevent
> overloading the bandwidth reservation of the merged stream and b) not
> drop packets due to congestion of the merge port, whether or not in-order
> delivery is desired.    If the merge buffer space is available, then there is no
> problem with full in-order delivery, and the user's misordering requirements
> are irrelevant.
> >>
> >> Finally, what do we imagine the network to do other than refuse the
> reservation in the very few cases where this is possible?  If it discards packets
> with larger differences, then you are doing nothing that the receiver cannot
> do for itself.  You also are taking on the problem of knowing when to resume
> transmission, detecting when the sequence number is reset, etc.  These are
> not trivial problems, and wind up needing help from the sequencing source.
> This is better left to the application to sort out.
> >>
> >> To sum up, I think that user requirements are much better expressed,
> and the network's ability to conform to those requirements are better
> measured, in terms of best- and worst-case latency and latency variation.
> >>
> >> -- Norm
> >> in-line with the recent developments in the WG. I'd like to have WG
> on
> >> which could be called "MisorderingTolerance" or alike to replace the
> >> "o An upper bound on out-of-order packet delivery. It is worth
> >> out-of-order delivery."
> numbers
> >> lowest sequence that is acceptable"