Subject: Re: [Detnet] updating the flow information model draft

From: János Farkas <[email-masked]>
Date: Wed, 17 Oct 2018 20:17:08 +0200

Hi,

Coming back to this.

In order delivery is one of the service parameters. As such, it reflects 
how much out of order delivery the application running in the end system 
can tolerate. If it is zero, then the application cannot tolerate 
misordering at all. A non-zero value indicates how many packets received 
out of order can be handled by the application.

Violation of service parameters implies that the application cannot 
handle (drops) the packets received. For instance, if the maximum 
tolerable end to end latency is violated, then the packet arrives too 
late for the application to be able to process. Similarly, if 
misordering is violated, the application cannot process the packet.

The service parameters are input to the DetNet network, which only 
accepts a request to establish a new flow if the network can meet the 
service parameters, otherwise rejects the request.

The comments received on the document in this discussion are actually on 
Section 5.2 Service parameters, e.g., delay and delay variation 
parameters, misordering etc.

In order to make progress, as a contribution to the WG, Section 5.2 will 
be updated and a new revision of the draft will be submitted by the 
submission deadline to address the comments to our best. The WG 
discussion can be then continued taking it as an input, and fruther 
updates will be made based on the discussions.

Best regards,
Janos

On 7/20/2018 8:50 PM, Norman Finn wrote:
> Lou:  Perhaps you could expand on your "different technologies" comment.
>
> Short explanation:
>
> The amount of misordering, and hence the resources (including choice of algorithm) required to re-order packets, is known the the network, not the listener.  The application already specifies a min/max latency in order to bound his receive buffer requirements.  I would favor 1) adding the ability of the network to report the worst-case misordering, 2) keeping the binary "application can/cannot tolerate misordering" parameter, 3) having a binary "network can/cannot ensure ordering".
>
> Long explanation:
>
> I stand by my arguments about 5/100 vs. no/yes.  If it's possible for the network to misorder packets, the resources required to reorder them, given the path and the bandwidth/packet size spec for the flow, is certainly known to the network, but probably not to the application.  Reporting the worst-case misordering to the application would be useful.  But, allowing the application to specify "5" when the network is likely to deliver 100 simply complicates the reservation process.  Of course, it's easy to compare the application's requirement to the network's number, but that is not the issue.  Having the parameter merely begs a pile of questions for the application:
>
>    - What number do I specify to maximize my chances of success?
>    - How many buffers might the network want me to have?
>    - Does that mean I need to know the topology?  Line lengths?
>    - How do I compare my need for receiver buffers to my other needs for memory space?
>    - Which device, me or the edge node, would find it easier/cheaper to increase the number of reordering buffers?
>    - Does the edge node even have reordering capability?
>    - Will the network configuration produce misordering, at all?
>
> My claim is that most of these questions are settled much more easily with the current yes/no requirement plus a report from the network on worst-case misordering, and we can see a path for negotiating/refusing/remaking the reservation.  I further claim that turning the yes/no into a number will complicate the process without bringing anything significant to the range of results available.
>
> For example, are we suggesting that the network could buffer just enough to reduce the 100 to 5?  I don't think it worthwhile to support the (I think infrequent) case where the listener's capabilities to handle some misordering would make a significant difference in the resources required for the network to reduce misordering to that level, but not eliminate it.
>
> I would rather not add a "misordering limit" parameter.  It will inevitably generate requirements for the control protocols support it, and I claim there is no clear use for that information.  Yes/no parameters for network and application capabilities, and a report on the possible misordering number from the network, is sufficient.
>
> -- Norm
>
> ________________________________________
> From: detnet [[email-masked]] on behalf of Lou Berger [[email-masked]]
> Sent: Thursday, July 19, 2018 7:27 AM
> To: Norman Finn; János Farkas; DetNet WG
> Subject: Re: [Detnet] updating the flow information model draft
>
> On 7/17/2018 7:01 PM, Norman Finn wrote:
>> Binary "reordering is/not acceptable" I can see some use for.  It's the numbers that I don't know how to deal with.
> There are different technology related/specific solutions here.  The
> fundamental point  is to allow for it in the architecture and modeling.
>
> Lou
>
>> -- Norm
>>
>> ________________________________________
>> From: Lou Berger [[email-masked]]
>> Sent: Tuesday, July 17, 2018 1:39 PM
>> To: Norman Finn; János Farkas; DetNet WG
>> Subject: Re: [Detnet] updating the flow information model draft
>>
>> Norm,
>>
>>        In the IP case, worst-case packet misordering can also impact
>> multi-path forwarding behavior.  An implementation (IP or MPLS data
>> planes) can always satisfy a non-zero reordering requirement with paths
>> that don't reorder.
>>
>> Lou
>>
>> On 7/17/2018 4:33 PM, Norman Finn wrote:
>>> I have a problem with the idea of the user expressing a requirement for worst-case packet misordering.  My problem is that I'm not sure what the network can do with that information.
>>>
>>> Either you are using PR/PE or not.  If you are NOT using PR/PE, then you are presumably using some routing/bridging protocol to heal the network after a failure.  The out-of-order problem will occur when, some time after a failure caused a loss of packets and the use of a long path A-->B, healing the failure and thus opening a shorter path causes early deliveries and hence out-of-order packets.  In order to predict the worst-case scenario for out-of-order delivery, so that the network can pass or fail the reservation request, one would have to determine the worst-case packets in flight for the longest path A-->B after multiple failures in the network, and then look at the fastest possible healing to the shortest possible path to get the worst-case out-of-order difference.  It might be possible to calculate this in a single-ring topology, but in a more complex topology, I have my doubts that it can be performed reliably.  Without a reliable way to calculate the worst case, the user's misordering requirements are irrelevant.
>>>
>>> If you ARE using PR/PE, then either you are using a bandwidth reservation or you are not.  If you are NOT using a bandwidth reservation, then all bets are off.  It is difficult to predict the worst-case difference in packets-in-flight for the two paths, and hence the worst-case sequence delta.  With nothing to compare the user's misording requirements to, the requirements are irrelevant.
>>>
>>> If you ARE using PR/PE and you ARE using bandwidth reservations, then as you are making the reservations, you can determine the worst-case time differential on the different paths before a merge point, you know the maximum data rates, and you can determine the buffer space needed to put the packets in order.    If the merge system has insufficient buffer space to accomplish the merge, then the user's misordering requirements are irrelevant.  That buffer space is required in order to both a) prevent overloading the bandwidth reservation of the merged stream and b) not drop packets due to congestion of the merge port, whether or not in-order delivery is desired.    If the merge buffer space is available, then there is no problem with full in-order delivery, and the user's misordering requirements are irrelevant.
>>>
>>> Finally, what do we imagine the network to do other than refuse the reservation in the very few cases where this is possible?  If it discards packets with larger differences, then you are doing nothing that the receiver cannot do for itself.  You also are taking on the problem of knowing when to resume transmission, detecting when the sequence number is reset, etc.  These are not trivial problems, and wind up needing help from the sequencing source.  This is better left to the application to sort out.
>>>
>>> To sum up, I think that user requirements are much better expressed, and the network's ability to conform to those requirements are better measured, in terms of best- and worst-case latency and latency variation.
>>>
>>> -- Norm
>>> ________________________________________
>>> From: detnet [[email-masked]] on behalf of János Farkas [[email-masked]]
>>> Sent: Wednesday, July 04, 2018 11:12 AM
>>> To: DetNet WG
>>> Subject: [Detnet] updating the flow information model draft
>>>
>>> Dear all,
>>>
>>> I think the flow information model draft needs some updates to bring it
>>> in-line with the recent developments in the WG. I’d like to have WG
>>> discussion on the list and at IETF 102 before making any updates.
>>>
>>> 1, Current version of draft-ietf-detnet-flow-information-model-01
>>>
>>> Section 5.2 Service Parameters includes in order delivery, which is a
>>> binary value in draft-ietf-detnet-flow-information-model-01 depending on
>>> whether or not in order delivery is required for the given service. This
>>> requires refinements based on the developments we had in
>>> draft-ietf-detnet-architecture-06.
>>>
>>> The refinement would involve introducing a numerical service parameter,
>>> which could be called “MisorderingTolerance” or alike to replace the
>>> current binary parameter.
>>>
>>>
>>> 2, Recent updates of draft-ietf-detnet-architecture-06
>>>
>>> draft-ietf-detnet-architecture-06 provides a finer grade DetNet QoS
>>> parameter by introducing :
>>>
>>> Section 3.1:
>>> “o An upper bound on out-of-order packet delivery. It is worth
>>> noting that some DetNet applications are unable to tolerate any
>>> out-of-order delivery.”
>>>
>>> Section 3.2.2.1. In-Order Delivery:
>>> “Out-of-order packet delivery can be a side effect of service
>>> protection. Packets delivered out-of-order impact the amount of
>>> buffering needed at the destination to properly process the received
>>> data. Such packets also influence the jitter of a flow. The DetNet
>>> service includes maximum allowed misordering as a constraint. Zero
>>> misordering would be a valid service constraint to reflect that the
>>> end system(s) of the flow cannot tolerate any out-of-order delivery.
>>> Service protection may provide a mechanism to support in-order
>>> delivery.”
>>>
>>> The architecture document does not provide further details on purpose.
>>> It is the job of the flow information model draft to go into the details.
>>>
>>>
>>> 3, “MisorderingTolerance” related thoughts
>>>
>>> The DetNet QoS requirements are correlated to each other. For instance,
>>> a packet received with a too big delay or jitter is practically a lost
>>> packet for the application. As for misordering, it is related, e.g., to
>>> jitter as explained in the architecture document. (Well, misordering
>>> above the tolerable limit can be considered loss.) The correlations can
>>> be taken into account when defining the QoS requirements for a DetNet
>>> service, and also when providing the given DetNet service by the DetNet
>>> network.
>>>
>>> Actually, misordering and jitter can be decreased with a similar tool,
>>> e.g., a playout buffer. It is a bit more complex for reordering and
>>> requires sequencing information carried in the packets.
>>>
>>> I see two approaches to express maximum allowed misordering:
>>> a) number of misordered packets
>>> b) time
>>>
>>> Both of them have their pros and cons.
>>>
>>> I propose to follow a) for multiple reasons.
>>>
>>> Maximum allowed/tolerable misordering could be defined along:
>>> “maximum number of misordered packets, e.g., a delta of sequence numbers
>>> between the highest sequence number that was just received and the
>>> lowest sequence that is acceptable”
>>> (credit goes to Pascal for initial wording!)
>>>
>>> This is a relatively simple definition and also simple to measure,
>>> detect, and verify. Furthermore, timing requirement is already defined
>>> as part of the jitter requirement, no need to duplicate. In addition,
>>> timing may not be as important as the order of packets for some
>>> applications.
>>>
>>> What do you think?
>>>
>>> Regards,
>>> Janos
>>>
>>> _______________________________________________
>>> detnet mailing list
>>> [email-masked]
>>> https://www.ietf.org/mailman/listinfo/detnet
>>>
>>> _______________________________________________
>>> detnet mailing list
>>> [email-masked]
>>> https://www.ietf.org/mailman/listinfo/detnet
>>>
>>
>
> _______________________________________________
> detnet mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/detnet

---

From: János Farkas <[email-masked]>
Date: Tue, 23 Oct 2018 00:21:21 +0200

Hi,

A new revision (02) of the  flow information model draft is available: 
https://tools.ietf.org/html/draft-ietf-detnet-flow-information-model-02.
As a contribution to the WG, it has been updated along the liens below. 
(plus a few nits related to references addressed)

Best regards,
Janos

On 10/17/2018 8:17 PM, János Farkas wrote:
> Hi,
>
> Coming back to this.
>
> In order delivery is one of the service parameters. As such, it 
> reflects how much out of order delivery the application running in the 
> end system can tolerate. If it is zero, then the application cannot 
> tolerate misordering at all. A non-zero value indicates how many 
> packets received out of order can be handled by the application.
>
> The comments received on the document in this discussion are actually 
> on Section 5.2 Service parameters, e.g., delay and delay variation 
> In order to make progress, as a contribution to the WG, Section 5.2 
> will be updated and a new revision of the draft will be submitted by 
> the submission deadline to address the comments to our best. The WG 
>> The amount of misordering, and hence the resources (including choice 
>> of algorithm) required to re-order packets, is known the the network, 
>> not the listener.  The application already specifies a min/max 
>> latency in order to bound his receive buffer requirements.  I would 
>> favor 1) adding the ability of the network to report the worst-case 
>> misordering, 2) keeping the binary "application can/cannot tolerate 
>> misordering" parameter, 3) having a binary "network can/cannot ensure 
>> ordering".
>>
>> Long explanation:
>>
>> I stand by my arguments about 5/100 vs. no/yes.  If it's possible for 
>> the network to misorder packets, the resources required to reorder 
>> them, given the path and the bandwidth/packet size spec for the flow, 
>> is certainly known to the network, but probably not to the 
>> application.  Reporting the worst-case misordering to the application 
>> would be useful.  But, allowing the application to specify "5" when 
>> the network is likely to deliver 100 simply complicates the 
>> reservation process.  Of course, it's easy to compare the 
>> application's requirement to the network's number, but that is not 
>> the issue..  Having the parameter merely begs a pile of questions for 
>> the application:
>>
>>    - Which device, me or the edge node, would find it easier/cheaper 
>> to increase the number of reordering buffers?
>> My claim is that most of these questions are settled much more easily 
>> with the current yes/no requirement plus a report from the network on 
>> worst-case misordering, and we can see a path for 
>> negotiating/refusing/remaking the reservation.  I further claim that 
>> turning the yes/no into a number will complicate the process without 
>> bringing anything significant to the range of results available.
>>
>> For example, are we suggesting that the network could buffer just 
>> enough to reduce the 100 to 5?  I don't think it worthwhile to 
>> support the (I think infrequent) case where the listener's 
>> capabilities to handle some misordering would make a significant 
>> difference in the resources required for the network to reduce 
>> misordering to that level, but not eliminate it.
>>
>> I would rather not add a "misordering limit" parameter.  It will 
>> inevitably generate requirements for the control protocols support 
>> it, and I claim there is no clear use for that information.  Yes/no 
>> parameters for network and application capabilities, and a report on 
>> the possible misordering number from the network, is sufficient.
>>
>> -- Norm
>>
>>> Binary "reordering is/not acceptable" I can see some use for.  It's 
>>> the numbers that I don't know how to deal with.
>>>> I have a problem with the idea of the user expressing a requirement 
>>>> for worst-case packet misordering.  My problem is that I'm not sure 
>>>> what the network can do with that information.
>>>>
>>>> Either you are using PR/PE or not.  If you are NOT using PR/PE, 
>>>> then you are presumably using some routing/bridging protocol to 
>>>> heal the network after a failure.  The out-of-order problem will 
>>>> occur when, some time after a failure caused a loss of packets and 
>>>> the use of a long path A-->B, healing the failure and thus opening 
>>>> a shorter path causes early deliveries and hence out-of-order 
>>>> packets.  In order to predict the worst-case scenario for 
>>>> out-of-order delivery, so that the network can pass or fail the 
>>>> reservation request, one would have to determine the worst-case 
>>>> packets in flight for the longest path A-->B after multiple 
>>>> failures in the network, and then look at the fastest possible 
>>>> healing to the shortest possible path to get the worst-case 
>>>> out-of-order difference.  It might be possible to calculate this in 
>>>> a single-ring topology, but in a more complex topology, I have my 
>>>> doubts that it can be performed reliably.  Without a reliable way 
>>>> to calculate the worst case, the user's misordering requirements 
>>>> are irrelevant.
>>>>
>>>> If you ARE using PR/PE, then either you are using a bandwidth 
>>>> reservation or you are not.  If you are NOT using a bandwidth 
>>>> reservation, then all bets are off.  It is difficult to predict the 
>>>> worst-case difference in packets-in-flight for the two paths, and 
>>>> hence the worst-case sequence delta.  With nothing to compare the 
>>>> user's misording requirements to, the requirements are irrelevant.
>>>>
>>>> If you ARE using PR/PE and you ARE using bandwidth reservations, 
>>>> then as you are making the reservations, you can determine the 
>>>> worst-case time differential on the different paths before a merge 
>>>> point, you know the maximum data rates, and you can determine the 
>>>> buffer space needed to put the packets in order.    If the merge 
>>>> system has insufficient buffer space to accomplish the merge, then 
>>>> the user's misordering requirements are irrelevant.  That buffer 
>>>> space is required in order to both a) prevent overloading the 
>>>> bandwidth reservation of the merged stream and b) not drop packets 
>>>> due to congestion of the merge port, whether or not in-order 
>>>> delivery is desired.    If the merge buffer space is available, 
>>>> then there is no problem with full in-order delivery, and the 
>>>> user's misordering requirements are irrelevant.
>>>>
>>>> Finally, what do we imagine the network to do other than refuse the 
>>>> reservation in the very few cases where this is possible?  If it 
>>>> discards packets with larger differences, then you are doing 
>>>> nothing that the receiver cannot do for itself.  You also are 
>>>> taking on the problem of knowing when to resume transmission, 
>>>> detecting when the sequence number is reset, etc.  These are not 
>>>> trivial problems, and wind up needing help from the sequencing 
>>>> source.  This is better left to the application to sort out.
>>>>
>>>> To sum up, I think that user requirements are much better 
>>>> expressed, and the network's ability to conform to those 
>>>> requirements are better measured, in terms of best- and worst-case 
>>>> latency and latency variation.
>>>>
>>>> -- Norm