Subject: Re: [Detnet] WG Last Call: draft-ietf-detnet-architecture-05

From: Balázs Varga A <[email-masked]>
Date: Mon, 11 Jun 2018 17:06:00 +0000

Hi David,

Many thanks for your great comments.
I am OK with the proposed resolution as well.

Your proposed resolution will be considered for
"Section 4.10.  Scaling to larger networks" as well.
However during the data plane discussions we have 
investigated more details on aggregation, therefore
changes will incorporate those results as well. We 
have discussed multiple options for aggregation
(not only classical tunneling). 

Thanks for your efforts.

Cheers
Bala'zs

-----Original Message-----
From: detnet <[email-masked]> On Behalf Of Black, David
Sent: 2018. május 25. 23:26
To: Lou Berger <[email-masked]>; DetNet WG <[email-masked]>
Cc: DetNet Chairs <[email-masked]>
Subject: Re: [Detnet] WG Last Call: draft-ietf-detnet-architecture-05

Looking at this from a Transport (Area) viewpoint, I have a few mostly editorial comments:

-- Section 3.1

   o  Minimum and maximum end-to-end latency from source to destination;
      timely delivery and jitter avoidance derive from these constraints

"jitter avoidance" -> "limits on jitter"
as jitter cannot be entirely avoided.

   o  Probability of loss of a packet, under various assumptions as to
      the operational states of the nodes and links.  A derived property
      is whether it is acceptable to deliver a duplicate packet, which
      is an inherent risk in highly reliable and/or broadcast transmissions

"derived" does not seem to be the right word in "derived property" and
the whole second sentence seems off.   Suggest:

   o  Probability of loss of a packet, under various assumptions as to
      the operational states of the nodes and links.  If packet replication
      is used to reduce the probability of packet loss, then a related
      property is the probability (may be zero) of delivery of a duplicate
      packet.  Duplicate packet delivery
      is an inherent risk in highly reliable and/or broadcast transmissions.

-- Section 3.2

Text in 3.1 lists three techniques used by DetNet to provide QoS, but section 3.2 (Mechanisms to achieve DetNet QoS) has five subsections.
An introductory paragraph in 3.2 (before 3.2.1) to explain how the five mechanisms relate to the three techniques would be helpful.  As 3.2.3 on jitter reduction is related to 3.2.1 on congestion protection, it might help to swap the order of 3.2.2 and 3.2.3 so that the congestion protection material is contiguous.

-- Section 4.1.1, after Figure 2

   The layers are, from top to bottom:

Not exactly - the list that follows does not contain an entry per layer.  Suggest

   The functionality shown in Figure 2 is:

and I would move OAM out of the list to its own paragraph at the end of Section 4.1.1, as OAM is not shown in Figure 2.

-- Section 4.3.2 Source guarantees

Section title seems off, as nothing is being guaranteed to the source.
Would "Source transmission behavior" or Source transmission requirements"
be clearer?

   The source promises that these limits will not be exceeded.

I don't understand "promises" - who/what is the recipient of the promise, and how is the promise made?  Suggest:

   The source is required not to exceed these limits in order to obtain DetNet service.

-- Section 4.3.3 Incomplete Networks

   as
   extra buffering, and thus extra latency, must be allocated at points
   downstream from the non-DetNet intermediate node for a DetNet flow.

I don't think "extra latency" is being "allocated" - in addition, jitter should be
mentioned.   Suggest:

   as
   extra buffering must be allocated at points
   downstream from the non-DetNet intermediate node for a DetNet flow.
   This extra buffering may increase latency and/or jitter.

-- Section 4.10.  Scaling to larger networks

   The DetNet data plane, in
   order to support larger numbers of DetNet flows, must support the
   aggregation of DetNet flows into tunnels, , which themselves can be
   viewed by the transit nodes' data planes largely as individual DetNet
   flows.

That sentence makes two points:
  a) Flow aggregation is required for scaling.
  b) That flow aggregation has to employ tunnels.

The first point is supported by the text in this section, but the second point does not appear to have supporting text.  It appears that tunnels are necessary
because: 
	- Each aggregated flow has to be a DetNet flow, and hence has to have
		its own 	DetNet flow ID.
	- The DetNet flow IDs of the flows that comprise an aggregated
		flow have to be carried through the aggregated flow in
		order to correctly demux to those original DetNet flows
		at the downstream end of flow aggregation.
A sentence or two should be added to explain this.

Thanks, --David
p.s.  I see that I'm already listed in the Acknowledgements section - that is appreciated.

> -----Original Message-----
> From: detnet [mailto:[email-masked]] On Behalf Of Lou Berger
> Sent: Friday, May 11, 2018 5:44 PM
> To: DetNet WG
> Cc: DetNet Chairs
> Subject: [Detnet] WG Last Call: draft-ietf-detnet-architecture-05
> 
> All,
> 
> This starts a two-week working group last call on
> draft-ietf-detnet-architecture-05
> 
> The working group last call ends on May 25.
> Please send your comments to the detnet mailing list.
> 
> Positive comments, e.g., "I've reviewed this document and believe it 
> is ready for publication", are welcome!
> This is useful and important, even from authors.
> 
> Thank you,
> DetNet Chairs
> 
> _______________________________________________
> detnet mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/detnet

_______________________________________________
detnet mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/detnet

---

From: Balázs Varga A <[email-masked]>
Date: Tue, 12 Jun 2018 09:33:56 +0000

Hi Stewart,

Thanks for the detailed review and the great comments.

My answers inline. Update of the document in progress.

@Norm: do you have a reference for network calculus?

@Co-authors: what to do with the HSR-PRP, ISA95 references?

Cheers

Bala'zs

-----Original Message-----
From: detnet <[email-masked]> On Behalf Of Stewart Bryant
Sent: 2018. május 23. 18:02
To: Lou Berger <[email-masked]>; DetNet WG <[email-masked]>
Cc: DetNet Chairs <[email-masked]>
Subject: Re: [Detnet] WG Last Call: draft-ietf-detnet-architecture-05

Some WG last call comments on draft-ietf-detnet-architecture

Best Regards

Stewart

=============

Abstract

   The capabilities can be managed by

   configuration, or by manual or automatic network management.

SB> I am not sure that network management (at least in the IETF is quite

SB> the right term. Most people think of what is described here as

SB> automatic network management as a routing protocol.

[Balázs Varga A] OK. I have deleted this sentence. Do not add to the abstract.

It is rather confusing.

===============

1.  Introduction

   Deterministic Networking (DetNet) is a service that can be offered by

   a network to DetNet flows.  DetNet provides these flows extremely low s/flows/flows with/ =========

   Many applications of interest to Deterministic Networking

SB> That is surely the wrong way round - the applications are interested

SB> in DN rather than DN being interested in the Apps?

[Balázs Varga A] OK. Fixed.

==========

   source

           An end system capable of sourcing a DetNet flow.

SB> Shouldn't that be a DetNet source, since you can have both DetNet

SB> and non-DetNet sources in a DetNet enabled network.

[Balázs Varga A] OK. Fixed

===========

    DetNet transit node

           A node operating at the DetNet transport layer, that utilizes

           link layer and/or network layer switching across multiple

           links and/or sub-networks to provide paths for DetNet service

           layer functions.  Optionally provides congestion protection

           over those paths.  An MPLS LSR is an example of a DetNet

           transit node.

SB> In that example it would have to be a DetNet enable/aware LSR. An

SB> ordinary LSR would not know anything about DetNet.

[Balázs Varga A] No, A DetNet aware LSR would be a relay node (S-PE).

=============

2.2.  IEEE 802 TSN to DetNet dictionary

   This section also serves as a dictionary for translating from the

   terms used by the IEEE 802 Time-Sensitive Networking (TSN) Task Group

   to those of the DetNet WG.

   Listener

           The IEEE 802 term for a destination of a DetNet flow.

   relay system

           The IEEE 802 term for a DetNet intermediate node.

   Stream

           The IEEE 802 term for a DetNet flow.

   Talker

           The IEEE 802 term for the source of a DetNet flow.

SB> Whilst I cannot comment on IEEE definitions, I am suprised that in

SB> our text a Listener is not called an 802 TSN Listener etc

[Balázs Varga A] We have used the TSN end-system. TSN end-system can be both

(Listener and Talker).

=============

   The paths are typically (but not necessarily) explicit

   routes, so that they cannot suffer temporary interruptions caused by

   the reconvergence of routing or bridging protocols.

SB> I think cannot is too strong. I think the best that you can say is

SB> "do not normally".

SB>

[Balázs Varga A] OK, Fixed.

============

   These three techniques can be applied independently, giving eight

   possible combinations, including none (no DetNet), although some

   combinations are of wider utility than others.  This separation keeps

   the protocol stack coherent and maximizes interoperability with

   existing and developing standards in this (IETF) and other Standards

   Development Organizations.  Some examples of typical expected

   combinations:

   o  Explicit routes plus service protection are exactly the techniques

      employed by [HSR-PRP].  Explicit routes are achieved by limiting

      the physical topology of the network, and the sequentialization,

      replication, and duplicate elimination are facilitated by packet

      tags added at the front or the end of Ethernet frames.

SB> ER can be done virtually as well as physically. RSVP is a type of

SB> ER, as is Adj-SID based SR, and we can design other types.

[Balázs Varga A] Agree, but these are examples. Statement is for HSR-PRP.

   o  Congestion protection alone is is offered by IEEE 802.1 Audio

      Video bridging [IEEE802.1BA-2011].  As long as the network suffers

      no failures, zero congestion loss can be achieved through the use

      of a reservation protocol (MSRP), shapers in every bridge, and a

      bit of network calculus.

SB> Network calculus needs a reference. It is not well known outside detnet.

[Balázs Varga A] Agree. Asked Norm for a reference.

================

3.2.3.  Jitter Reduction

   A core objective of DetNet is to enable the convergence of Non-IP

   networks onto a common network infrastructure.

SB> I think that  this needs to be clarified that it is the convergence

SB> of sensitive Non-IP. PW has been doing this convergence since

SB> inception, and can handle what some would expect to be sensitive

SB> applications such as TDM telephony quite well.

[Balázs Varga A] OK. Fixed.

   This requires the

   accurate emulation of currently deployed mission-specific networks,

   which typically rely on point-to-point analog (e.g. 4-20mA

   modulation) and serial-digital cables (or buses) for highly reliable,

   synchronized and jitter-free communications.  While the latency of

   analog transmissions is basically the speed of light, legacy serial

   links are usually slow (in the order of Kbps) compared to, say, GigE,

   and some latency is usually acceptable.  What is not acceptable is

   the introduction of excessive jitter, which may, for instance, affect

   the stability of control systems.

SB> See above, I think this needs a few caveats.

[Balázs Varga A] OK. Fixed.

===============

   Packet replication and elimination, also known as seamless redundancy

   [HSR-PRP], or 1+1 hitless protection, is a function of the DetNet

   service layer.  It involves three capabilities:

SB> I am concerned at the prominence of the HSR-PRP reference. As far as

SB> I can tell this is only available for fee of 300 Euro.

[Balázs Varga A] OK. Let's discuss this with the co-authors.

============

   In the simplest case, this amounts to replicating each packet in a

   source that has two interfaces, and conveying them through the

   network, along separate paths, to the similarly dual-homed

   destinations, that discard the extras.  This ensures that one path

   (with zero congestion loss) remains, even if some intermediate node

   fails.  The sequence numbers can also be used for loss detection and

   for re-ordering.

SB> Sure, but of course you need to do SRLG risk analysis, and even then

SB> two failures are not unknowm.

[Balázs Varga A] OK. Fixed.

===========

                    Packet replication and elimination

                > > > > > > > > > relay > > > > > > > >

               > /------------+ R node E +------------\ >

              > /                  v + ^               \ >

      end    R +                   v | ^                + E end

      system   +                   v | ^                +   system

              > \                  v + ^               / >

               > \------------+ R relay E +-----------/ >

                > > > > > > > > >  node > > > > > > > >

                                 Figure 1

   Packet replication and elimination does not react to and correct

   failures; it is entirely passive.  Thus, intermittent failures,

SB> I think it copes with intermittent failures OK.

[Balázs Varga A] Yes, PRF and PEF can eliminate the effect of such failures. But text

intends to say that it is passive. It is not reacting to such failures. No change in text.

    mistakenly created packet filters, or misrouted data is handled just

   the same as the equipment failures that are detected handled by

   typical routing and bridging protocols.

================

   Transit nodes see DetNet nodes as end points.  All

   DetNet enabled nodes are connect to sub-networks, where a point-to-

SB> s/connect/connected/

[Balázs Varga A] OK. Fixed.

===========

   o  DetNet t-aware: An extant TSN system.  It knows about some TSN

      functions (e.g., reservation), but not about replication/

      elimination.

SB> Generic TSN is OK, but I would be cautious of requiring IEEE TSN.

[Balázs Varga A] OK.

===========

 o  DetNet s-aware: An extant IEC 62439-3 system.  It supplies

      sequence numbers, but doesn't know about zero congestion loss.

SB> IEC62439 should surely be clarified as just an example.

[Balázs Varga A] OK. All items are examples " Note some known use cases for end systems: "

===========

4.3.1.  DetNet flow types

   A DetNet flow can have different formats during while it is

SB> s/during while/as/

[Balázs Varga A] OK. Fixed.

===========

   transported between the peer end systems.  Therefore, the following

   possible types / formats of a DetNet flow are distinguished in this

   document:

   o  App-flow: native format of a DetNet flow.  It does not contain any

      DetNet related attributes.

   o  DetNet-t-flow: specific format of a DetNet flow.  Only requires

      the congestion / latency features provided by the Detnet transport

      layer.

   o  DetNet-s-flow: specific format of a DetNet flow.  Only requires

      the replication/elimination feature ensured by the DetNet service

      layer.

   o  DetNet-st-flow: specific format of a DetNet flow.  It requires

      both DetNet service layer and DetNet transport layer functions

      during forwarding.

SB> I find the relisting of these types confusing. Wheren't they defined

SB> in the section above?

[Balázs Varga A] This is inline with the previous section " Grouping of end systems ".

===========

4.3.2.  Source guarantees

   For the purposes of congestion protection, DetNet flows can be

   synchronous or asynchronous.  In synchronous DetNet flows, at least

   the intermediate nodes (and possibly the end systems) are closely

   time synchronized, typically to better than 1 microsecond.

SB> Are you talking about the flows being sync, or the network handling

of the

SB> the flows here. We have carried synchronous traffic over MPLS in the

past

SB> so I think this is a topic that requires quite careful handling to

make sure that

SB> we do not set up an over-engineered solution.

[Balázs Varga A] "Sync as a service" is assumed for some use-case.

It is not needed for some others.

==========

   The source promises that these limits will not be exceeded.  If the

   source transmits less data than this limit allows, the unused

   resources such as link bandwidth can be made available by the system

SB> s/can be made/to be made/

[Balázs Varga A] OK. Fixed.

==========

4.4.  Traffic Engineering for DetNet

   Traffic Engineering Architecture and Signaling (TEAS) [TEAS] defines

   traffic-engineering architectures for generic applicability across

   packet and non-packet networks.  From TEAS perspective, Traffic

SB> s/from/from a/

[Balázs Varga A] OK Fixed.

==========

   ..... to translate from a flow

   specification to a set of values for the managed objects that control

   each relay or end system.  The IEEE 802 has specified (and is

SB> Should that be IEEE 802 Standards Group or some such?

[Balázs Varga A] OK. Fixed.

=============

   The DetNet network reference model is shown in Figure 8 for a DetNet-

   Service scenario (i.e. between two DetNet-UNIs).  In this figure, the

   end systems ("A" and "B") are connected directly to the edge nodes of

   the IP/MPLS network ("PE1" and "PE2").

SB> I am confused as to when we should use the terms "relay node", "edge

node"

SB> and "PE". There seem.s to be inconsistency in the project on the use

SB> of these terms

[Balázs Varga A] OK. See separate mailing started by Lou. "PEx" here refers

to the nodes on the figure.

===========

   The tunnel between the service instances may have some special

   characteristics.  For example, in case of a "packet PW" based tunnel,

SB> I don't think we should use the term "packet PW" since that points

SB> to a specific PW technology and that is not the direction we are

SB> going in the data-plane part of the project.

SB> I think the text above and the text in the rest of the para is

SB> confusing for the reader.

SB> I suggest that we write something much more generic and let the DP

SB> team have a clear hand to design a good solution.

[Balázs Varga A] OK. Fixed. I have removed the " packet PW for DetNet traffic " term.

I agree that such a term is now outdated with all the data plane discussions behind us.

Text intention is to highlight the differences to rfc6658, so I have changed text to refer

"DetNet L3 service" scenario.

   there are differences in the usage of the packet PW for DetNet

   traffic compared to the network model described in [RFC6658]. In the

   DetNet scenario, the packet PW is used exclusively by the DetNet

   flow, whereas [RFC6658] states: "The packet PW appears as a single

   point-to-point link to the client layer.  Network-layer adjacency

   formation and maintenance between the client equipments will follow

   the normal practice needed to support the required relationship in

   the client layer ... This packet pseudowire is used to transport all

   of the required layer 2 and layer 3 protocols between LSR1 and LSR2".

===========

   The need for a lower-level DetNet node to be aware of individual

   higher-layer flows is not unique to DetNet.  But, given the endless

   complexity of layering and relayering over tunnels that is available

   to network designers, DetNet needs to provide a model for flow

   identification that is at least somewhat better than packet

SB> "at least somewhat better is a strange turn of phrase"

[Balázs Varga A] OK. Fixed.

===========

   Thus, rather than packet inspection, there is the option to export

   higher-layer information to the lower layer.  The requirement to

   support one or the other method for flow identification (or both) is

   the essential complexity that DetNet brings to existing control plane

   models.

SB> A very strange way of expressing the point, sounds like complexity

SB> is a virtue.

[Balázs Varga A] OK. Fixed. (essential --> necessary)

===========

Flow-IDs

   The additional (domain specific) Flow-ID can be

   o  created by a domain specific function or

   o  derived from the Flow-ID added to the App-flow,

   so that it must be unique inside the given domain.

SB> very strange English.

[Balázs Varga A] OK. Fixed (at least I hope so ...)

   Note that the

   Flow-ID added to the App-flow is still present in the packet, but

   transport nodes may lack the function to recognize it; that's why the

   additional Flow-ID is added (pushed).

SB> Again rather strange English and an unnecessary pointer to a

SB> particular implementation technique (pushed). Same comment applies

SB> to para 1 of

4.7.3

[Balázs Varga A] OK. Fixed (at least I hope so ...)

============

4.9.  Provisioning model

SB> Sections 4.9.1 and 4.9.2 seem rather tentative for an RFC that we

SB> are about to publish.

[Balázs Varga A] Agree. Historically that was a placeholder for topics we need further discussions.

I have deleted 4.9.

4.9.1.  Centralized Path Computation and Installation

4.9.2.  Distributed Path Setup

==============

5.  Security Considerations

   Furthermore, in a control system where millions of dollars of

   equipment, or even human lives, can be lost if the DetNet QoS is not

   delivered, one must consider not only simple equipment failures,

   where the box or wire instantly becomes perfectly silent, but bizarre

   errors such as can be caused by software failures.

SB> Bizare is a strange term to use. I am sure that later reviewers will

SB> ask for more text on this.

SB> I might use the term "subtle, complex, intermittent and state dependent"

[Balázs Varga A] OK. Fixed.

============

   [HSR-PRP]  IEC, "High availability seamless redundancy (HSR) is a

              further development of the PRP approach, although HSR

              functions primarily as a protocol for creating media

              redundancy while PRP, as described in the previous

              section, creates network redundancy.  PRP and HSR are both

              described in the IEC 62439 3 standard.",

              <http://webstore.iec.ch/webstore/webstore.nsf/

              artnum/046615!opendocument>.

SB> Not available at the time of this review, but my recollection is

SB> that this is not readily available without paying a large fee.

===========

   [ISA95]    ANSI/ISA, "Enterprise-Control System Integration Part 1:

              Models and Terminology", 2000,

              <https://www.isa.org/isa95/>.

SB> Should that be 2000, or 2010.

SB> Note that this seems to be a very expensive document to access.

=========

detnet mailing list

[email-masked]<mailto:[email-masked]>

---

From: "Black, David" <[email-masked]>
Date: Tue, 12 Jun 2018 15:27:39 +0000

Hi Bala'zs,

> I am OK with the proposed resolution as well.

Great!

> (not only classical tunneling).

That's fine - my overall sense was that the text involved would benefit some deeper thinking.  That appears to have happened in the data plane discussions, so reflecting the results of those discussions in revising that text sounds like a fine thing to do.  I look forward to seeing the result.

Thanks, --David

> From: Balázs Varga A [mailto:[email-masked]]
> Sent: Monday, June 11, 2018 1:06 PM
> To: Black, David; Lou Berger; DetNet WG
> Cc: DetNet Chairs
> Subject: RE: [Detnet] WG Last Call: draft-ietf-detnet-architecture-05
> 
> Hi David,
> 
> Text in 3.1 lists three techniques used by DetNet to provide QoS, but section
> 3.2 (Mechanisms to achieve DetNet QoS) has five subsections.
> An introductory paragraph in 3.2 (before 3.2.1) to explain how the five
> mechanisms relate to the three techniques would be helpful.  As 3.2.3 on
> jitter reduction is related to 3.2.1 on congestion protection, it might help to
> swap the order of 3.2.2 and 3.2.3 so that the congestion protection material is
> contiguous.
> 
> and I would move OAM out of the list to its own paragraph at the end of
> Section 4.1.1, as OAM is not shown in Figure 2.
> 
> The first point is supported by the text in this section, but the second point
> does not appear to have supporting text.  It appears that tunnels are
> necessary
> because:

---

From: Stewart Bryant <[email-masked]>
Date: Tue, 12 Jun 2018 17:27:13 +0100

Dear Bala'zs

Thank you your for your consideration of these points. I will just pick 

> [Balázs Varga A] No, A DetNet aware LSR would be a relay node (S-PE).

I think the confusion is what "DetNet Transport Layer" means. This
technology touches on Transport Layer in  the L4 sense, and the
Transport Network Layer as in the packet network that carries
L3 packets.

So I think that we need to clarify whether a DetNet transit node
is an S-PE (i.e. a a transit node in the DetNet layer), or a P node
(i.e. a transit node that is carrying DetNet packets but could be
carrying any sort of MPLS packet)

So the question is whether we should expand the set of examples, 
particularly
to more accessible ones.

============
>                 > > > > > > > > > relay > > > > > > > >
>
>                > /------------+ R node E +------------\ >
>
>               > /                  v + ^               \ >
>
>       end    R +                   v | ^                + E end
>
>       system   +                   v | ^                +   system
>
>               > \                  v + ^               / >
>
>                > \------------+ R relay E +-----------/ >
>
>                 > > > > > > > > >  node > > > > > > > >
>
>                                  Figure 1
>

You say that PREF does not correct failures. I would contend that is exactly
what it does. Sure it does not react but it does correct, and it does
address intermittent failures.
Surely if we have defined them we never need to do anything but name them in
later sections. Redefinition is never a good idea because it often leads to
conflicting definitions.

If we decide that this is essential as a key reference, there needs to be
some suitable text, as this will get raised a number of times between
here an publication as first the directorates and then the ADs run
into this.

You did not comment on the correctness of the reference.

Best Regards

Stewart

---

From: János Farkas <[email-masked]>
Date: Fri, 29 Jun 2018 03:08:47 +0200

Dear all,

Off-line discussions among Lou, Stewart, and authors followed the 
discussions to properly address the WGLC comments, including the 
detailed comments.

A new revision of the draft has been uploaded: 
draft-ietf-detnet-architecture-06.

In addition to the changes already described in this thread, the 
following bigger changes have been made to the draft:

*Section 2.1 Terms used in this document*

Some definitions refined as suggested by the detailed comments

New definitions have been added:

    "allocation
            Resources are dedicated to support a DetNet flow. Depending
            on an implementation, the resource may be reused by non-
            DetNet flows when it is not used by the DetNet flow.

    PEF     A Packet Elimination Function (PEF) eliminates duplicate
            copies of packets to prevent excess packets flooding the
            network or duplicate packets being sent out of the DetNet
            domain.  PEF can be implemented by an edge node, a relay
            node, or an end system.

   PRF     A Packet Replication Function (PRF) replicates DetNet flow
            packets and forwards them to one or more next hops in the
            DetNet domain.  The number of packet copies sent to each next
            hop is a DetNet flow specific parameter at the node doing the
            replication.  PRF can be implemented by an edge node, a relay
            node, or an end system.

    PREOF   Collective name for Packet Replication, Elimination, and
            Ordering Functions.

    POF     A Packet Ordering Function (POF) re-orders packets within a
            DetNet flow that are received out of order.  This function
            can be implemented by an edge node, a relay node, or an end
            system.

   DetNet service proxy
            Maps between App-flows and DetNet flows.

    bridged path
            A VLAN bridge uses the VLAN ID and the destination MAC
            address to select the outbound port hence the path for a
            frame."

*Section 3.1 Primary goals defining the DetNet QoS*

A new QoS aspect has been added:
    "o  An upper bound on out-of-order packet delivery.  It is worth
       noting that some DetNet applications are unable to tolerate any
       out-of-order delivery."

The 3rd paragraph on loss on page 8 after the bullet list has been 
extended to:
    "After congestion, the most important contributions to packet loss are
    typically from random media errors and equipment failures. Service
    protection is the name for the mechanisms used by DetNet to address
    these losses.  The mechanisms employed are constrained by the
    requirement to meet the users' latency requirements.  Packet
    replication and elimination (Section 3.2.2) and packet encoding
    (Section 3.2.2.3) are described in this document to provide service
    protection; others may be found.  For instance, packet encoding can
    be used to provide service protection against random media errors,
    packet replication and elimination can be used to provide service
    protection against equipment failures.  This mechanism distributes
    the contents of DetNet flows over multiple paths in time and/or
    space, so that the loss of some of the paths does need not cause the
    loss of any packets."

*3.2.2.  Service Protection*

Service protection is used as a more generic term. Introductory text added:
    "Service protection aims to mitigate or eliminate packet loss due to
    equipment failures, random media and/or memory faults.  These types
    of packet loss can be greatly reduced by spreading the data over
    multiple disjoint forwarding paths.  Various service protection
    methods are described in [RFC6372], e.g., 1+1 linear protection.
    This section describes the functional details of an additional method
    in Section 3.2.2.2, which can be implemented as described in
    Section 3.2.2.3 or as specified in [I-D.ietf-detnet-dp-sol-mpls] in
    order to provide 1+n hitless protection.  The appropriate service
    protection mechanism depends on the scenario and the requirements."

New sub-section added:

"3.2.2.1.  In-Order Delivery

    Out-of-order packet delivery can be a side effect of service
    protection.  Packets delivered out-of-order impact the amount of
    buffering needed at the destination to properly process the received
    data.  Such packets also influence the jitter of a flow.  The DetNet
    service includes maximum allowed misordering as a constraint. Zero
    misordering would be a valid service constraint to reflect that the
    end system(s) of the flow cannot tolerate any out-of-order delivery.
    Service protection may provide a mechanism to support in-order
    delivery."

*3.2.2.2. Packet Replication and Elimination*

New bullet added as the last one:
    "o  The Packet Ordering Function (POF) uses the sequencing information
       to re-order a DetNet flow's packets that are received out of
       order."

New sentence added after the bullet list:
"The order in which a node applies the PEF and the PRF to a DetNet
flow is implementation specific."

2nd paragraph after the bullet list has been updated to:
"Some service protection mechanisms rely on switching from one flow to
    another when a failure of a flow is detected.  Contrarily, packet
    replication and elimination combines the DetNet member flows sent
    along multiple different paths, and performs a packet-by-packet
    selection of which to discard, e.g., based on sequencing information."

*3.2.3.  Explicit routes*

Out-of-order aspect added to the first paragraph, which is about 
distributed routing:
"Furthermore, out-of-order
packet delivery can be a side effect of route changes."

New sentence added to the 3rd paragraph:
"Explicit routes can be established various
ways, e.g., with RSVP-TE [RFC3209], with Segment Routing (SR)
[I-D.ietf-spring-segment-routing], via a Software Defined Networking
approach [RFC7426], with IS-IS [RFC7813], etc."

New paragraph added:
    "Out-of-order packet delivery can be a side effect of distributing a
    single flow over multiple paths especially when there is a change
    from one path to another when combining the flow.  This is
    irrespective of the distribution method used, also applies to service
    protection over explicit routes.  As described in Section 3.2.2.1,
    out-of-order packets influence the jitter of a flow and impact the
    amount of buffering needed to process the data; therefore, DetNet
    service includes maximum allowed misordering as a constraint. The
    use of explicit routes helps to provide in-order delivery because
    there is no immediate route change with the network topology, but the
    changes are plannable as they are between the different explicit
    routes."

*
**4.1.1. Representative Protocol Stack Model*

"Explicit routes" have been added to Figure 2 with the corresponding 
explanation:
  "Explicit routes
            The DetNet transport layer provides mechanisms to ensure that
            fixed paths are provided for DetNet flows.  These explicit
            paths avoid the impact of network convergence."

Section 4.11 Connected islands vs. networks of v05 has been deleted 
because it was just a leftover from early drafts on what DetNet WG 
should do; which are covered by the charter anyways.

References have been cleaned up and brought up-to-date.

Refinements have been implemented in the draft according to Lou's 
detailed comments. They are not listed here because they are minor changes.

Best regards,
Janos

On 6/12/2018 2:48 PM, Lou Berger wrote:
>
> Balázs,
>
> Thanks for the response -- please see below.
>
> ----------
> On June 12, 2018 4:07:35 AM Balázs Varga A 
> <[email-masked]> wrote:
>
>> Hi Lou, Thanks for the comments. See reactions inline. Document 
>> update in progress. Cheers Bala'zs
>>
>> From: detnet <[email-masked]> On Behalf Of Lou Berger
>> Sent: 2018. június 1. 22:42
>> To: DetNet WG <[email-masked]>; [email-masked]
>> Subject: [Detnet] Promised comments on draft-ietf-detnet-architecture
>>
>> Hi,
>>
>>      I have a number of high level comments on the document that I'd 
>> like to  raise below.  I also have a number of more 
>> editorial/specific comments that  I'd like to review directly with 
>> the authors and then have them report back  on changes -- if any turn 
>> out to be more substantive discussions from the  author's 
>> perspective, I'll raise these on the list separately.
>>
> ...
>>
>> - WRT Section 4.4.3, I think the inclusion of a distributed control 
>> plane in the "Network Plane" is inconsistent with other functional 
>> definitions and conflates where a function resides from the actual 
>> function and that whether control is implemented in a fully 
>> centralized, fully distributed or some hybrid form doesn't 
>> fundamentally change  the control function's role -- therefore I 
>> think the sections 4.4.2 and .3 should be revised accordingly
>> [Balázs Varga A] Agree in principal. Let's discuss the details.
>>
> Okay - I'll work with you off line and we can report back the 
> results/proposed changes.
>
>> - In several places it's not clear that DetNet service is always a L3 
>> service which is controlled using L3 identifiers, i.e., IP addresses 
>> -- this is true even in the MPLS service case and the TSN over MPLS 
>> case. I think this is an important point to be clear on in the document.
>> [Balázs Varga A] I am not sure. DetNet service is always provided 
>> over a L3 network (IP or MPLS), that is fine. However the service 
>> itself can be L2 or L3. In case of TSN Ethernet frames are 
>> transported, so it is a L2 service. In case of IP end systems IP 
>> packets are transported so it is a L3 service.
>>
>
> Humm - While I agree that DetNet is providing an (enhanced) L2VPN 
> service, it is not itself providing control or service of L2 devices 
> -- this is TSN's job.  The fact that DetNet is all about behavior of 
> L3 nodes (i.e., IP and PW/MPLS) and not L2 nodes (i.e., TSN bridges) 
> is something the architecture should make unambiguous.
>
> Thanks,
> Lou
>
>> Please let me know what you think.
>>
>> Cheers,
>>
>> Lou
>>
>>

On 6/12/2018 6:27 PM, Stewart Bryant wrote:
>
> Dear Bala'zs
>
> Thank you your for your consideration of these points. I will just 
> pick up a few that need some further thought:
>
>
>>     DetNet transit node
>>

---

From: Lou Berger <[email-masked]>
Date: Thu, 28 Jun 2018 22:35:08 -0400

Authors,

     Thank you for the update!

WG,

     This document has changed to a sufficient degree that I think a 2nd 
last call is warranted.  Typically I would start a 1 week LC in these 
circumstances - but given the proximity to the meeting I'd like to start 
a 3 week LC right ending with the IETF meeting -- that is on July 20th.  
This should allow for both adequate review of the changes and discussion 
of the changes in our WG session (Monday July 16.)

As always, please send LC comment to the list and positive comments, 
e.g., "I've reviewed this document
and believe it is ready for publication", are welcome!

Thank you,

Lou (as Shepherd)

On 6/28/2018 9:08 PM, János Farkas wrote:
>>> the authors and then have them report back  on changes -- if any 
>>> turn out to be more substantive discussions from the  author's 
>>> - In several places it's not clear that DetNet service is always a 
>>> L3 service which is controlled using L3 identifiers, i.e., IP 
>>> addresses -- this is true even in the MPLS service case and the TSN 
>>> over MPLS case. I think this is an important point to be clear on in 
>>> the document.

---

From: "Andrew G. Malis" <[email-masked]>
Date: Fri, 29 Jun 2018 09:05:10 -0400

Lou,

I've reviewed this document and believe it is ready for publication.

Cheers,
Andy

On Thu, Jun 28, 2018 at 10:35 PM, Lou Berger <[email-masked]> wrote:

> circumstances - but given the proximity to the meeting I'd like to start a
> 3 week LC right ending with the IETF meeting -- that is on July 20th.  This
> should allow for both adequate review of the changes and discussion of the
> changes in our WG session (Monday July 16.)
>
> As always, please send LC comment to the list and positive comments, e.g.,
> "I've reviewed this document
>> discussions to properly address the WGLC comments, including the detailed
>> comments.
>>
>> because it was just a leftover from early drafts on what DetNet WG should
>> do; which are covered by the charter anyways.
>>
>>
>>> On June 12, 2018 4:07:35 AM Balázs Varga A <[email-masked]>
>>> wrote:
>>>
>>> Hi Lou, Thanks for the comments. See reactions inline. Document update
>>>> in progress. Cheers Bala'zs
>>>>
>>>> like to  raise below.  I also have a number of more editorial/specific
>>>> comments that  I'd like to review directly with the authors and then have
>>>> them report back  on changes -- if any turn out to be more substantive
>>>> discussions from the  author's perspective, I'll raise these on the list
>>>> separately.
>>>>
>>>> ...
>>>
>>>>
>>>> definitions and conflates where a function resides from the actual function
>>>> and that whether control is implemented in a fully centralized, fully
>>>> distributed or some hybrid form doesn't fundamentally change  the control
>>>> function's role -- therefore I think the sections 4.4.2 and .3 should be
>>>> revised accordingly
>>>> service which is controlled using L3 identifiers, i.e., IP addresses --
>>>> this is true even in the MPLS service case and the TSN over MPLS case. I
>>>> think this is an important point to be clear on in the document.
>>>> [Balázs Varga A] I am not sure. DetNet service is always provided over
>>>> a L3 network (IP or MPLS), that is fine. However the service itself can be
>>>> L2 or L3. In case of TSN Ethernet frames are transported, so it is a L2
>>>> service. In case of IP end systems IP packets are transported so it is a L3
>>>> service.
>>>>
>>>>
>>> service, it is not itself providing control or service of L2 devices --
>>> this is TSN's job.  The fact that DetNet is all about behavior of L3 nodes
>>> (i.e., IP and PW/MPLS) and not L2 nodes (i.e., TSN bridges) is something
>>> the architecture should make unambiguous.
>>>
>>> Thanks,
>>> Lou
>>>

---

From: "Black, David" <[email-masked]>
Date: Fri, 29 Jun 2018 14:10:06 +0000

The -06 version of this draft is much improved over the -05 version – lots of diligent work by the authors is evident and appreciated.

I have two minor content concerns and some smaller editorial items.

** Content

-- Section 2.2

The definition of “bridged path” looks like a definition of “bridged forwarding” – the definition needs to encompass the notion of a path that involves more than one bridge.

   bridged path

           frame.

-- Section 4

Figure 2 in Section 4.1.1 introduces the Detnet service and transport layers.   The associated text quickly dives into explaining the functionality in those layers, but it would be helpful to start with an initial high level description of each of those layers that helps motivate their separation.  It appears that:
- The DetNet Service layer provides timely reliable in-order DetNet flow delivery to applications that use DetNet.   A DetNet flow at the top of the DetNet Service layer may or may not be a compound flow at the bottom of that layer.
- The DetNet Transport layer insulates DetNet flows (at the bottom of the DetNet Service layer) from interference by network mechanisms, primarily queuing and routing, where congestion is an indication of queueing interference.  A DetNet flow at the top of the DetNet Transport layer may or may not be a member flow that is part of a compound flow, depending on what is done in the DetNet Service layer.

There’s a related paragraph of text between Figures 3 and 4 in Section 4.1.2, but it doesn’t provide useful intuition about which functionality belongs in which layer.

** Editorial

-- Section 2.1

“expected” seems off in the definition of reservation:

   reservation
           The set of resources allocated between a source and one or
           more destinations through transit nodes and subnets
           associated with a DetNet flow, to provide the expected DetNet
           Service.

Suggest either “to provide the provisioned DetNet Service to that flow” or “to provide the specified DetNet Service to that flow.”

-- Section 3.1

   o  An upper bound on out-of-order packet delivery.  It is worth

      out-of-order delivery.

Is that a sequence bound or a time bound or both?   I suspect that a sequence bound (maximum degree of reordering in a flow, independent of flow rate) was intended.  An analogous clarification would be helpful in section 3.2.2.1 .

-- Section 3.2.3

I don’t understand “defined” in:

   Even the use of

   redundant paths through a network defined, e.g., by [RFC6372] do not

   eliminate the chances of packet loss.

Was “e.g., as defined by [RFC6372],” intended?

   Many real-time networks rely on physical rings or chains of two-port

   devices, with a relatively simple ring control protocol.

How does a “ring control protocol” control “chains”?  Suggest: “ring control protocol” -> “control protocol”
Also, are paired chains between common endpoints intended, or a more general notion of chains?

   Explicit routes can be established various ways,

established various -> established in various

   This is

   protection over explicit routes.

used, also -> used, and also

-- Figure 5 (Section 4.1.3)

The UNI acronym is introduced without expansion or definition – please correct that, e.g., by defining UNI in Section 2.1.

-- Figure 9 (Section 4.7.2)

I think I understand what’s going on, but the use of “add/remove” in the figure should be explained in associated text.

Thanks, --David

From: detnet [mailto:[email-masked]] On Behalf Of János Farkas
Sent: Thursday, June 28, 2018 9:09 PM
To: Lou Berger; Stewart Bryant; [email-masked] >> DetNet WG
Subject: Re: [Detnet] WG Last Call: draft-ietf-detnet-architecture-05

Dear all,

Off-line discussions among Lou, Stewart, and authors followed the discussions to properly address the WGLC comments, including the detailed comments.

A new revision of the draft has been uploaded: draft-ietf-detnet-architecture-06.

In addition to the changes already described in this thread, the following bigger changes have been made to the draft:

Section 2.1 Terms used in this document

   "allocation

  DetNet service proxy

   bridged path

Section 3.1 Primary goals defining the DetNet QoS

The 3rd paragraph on loss on page 8 after the bullet list has been extended to:

3.2.2.  Service Protection

New sub-section added:

3.2.2.2. Packet Replication and Elimination

3.2.3.  Explicit routes

Out-of-order aspect added to the first paragraph, which is about distributed routing:
"Furthermore, out-of-order
packet delivery can be a side effect of route changes."

New paragraph added:

4.1.1. Representative Protocol Stack Model

"Explicit routes" have been added to Figure 2 with the corresponding explanation:
 "Explicit routes
           The DetNet transport layer provides mechanisms to ensure that
           fixed paths are provided for DetNet flows.  These explicit
           paths avoid the impact of network convergence."

Section 4.11 Connected islands vs. networks of v05 has been deleted because it was just a leftover from early drafts on what DetNet WG should do; which are covered by the charter anyways.

Refinements have been implemented in the draft according to Lou's detailed comments. They are not listed here because they are minor changes.

Best regards,
Janos

Balázs,

----------
On June 12, 2018 4:07:35 AM Balázs Varga A <[email-masked]><mailto:[email-masked]> wrote:

Hi Lou, Thanks for the comments. See reactions inline. Document update in progress. Cheers Bala'zs

-----Original Message-----
From: detnet <[email-masked]><mailto:[email-masked]> On Behalf Of Lou Berger
Sent: 2018. június 1. 22:42
To: DetNet WG <[email-masked]><mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: [Detnet] Promised comments on draft-ietf-detnet-architecture

Hi,

     I have a number of high level comments on the document that I'd like to  raise below.  I also have a number of more editorial/specific comments that  I'd like to review directly with the authors and then have them report back  on changes -- if any turn out to be more substantive discussions from the  author's perspective, I'll raise these on the list separately.
...

- WRT Section 4.4.3, I think the inclusion of a distributed control plane in the "Network Plane" is inconsistent with other functional definitions and conflates where a function resides from the actual function and that whether control is implemented in a fully centralized, fully distributed or some hybrid form doesn't fundamentally change  the control function's role -- therefore I think the sections 4.4.2 and .3 should be revised accordingly
[Balázs Varga A] Agree in principal. Let's discuss the details.
Okay - I'll work with you off line and we can report back the results/proposed changes.

- In several places it's not clear that DetNet service is always a L3 service which is controlled using L3 identifiers, i.e., IP addresses -- this is true even in the MPLS service case and the TSN over MPLS case. I think this is an important point to be clear on in the document.
[Balázs Varga A] I am not sure. DetNet service is always provided over a L3 network (IP or MPLS), that is fine. However the service itself can be L2 or L3. In case of TSN Ethernet frames are transported, so it is a L2 service. In case of IP end systems IP packets are transported so it is a L3 service.

Humm - While I agree that DetNet is providing an (enhanced) L2VPN service, it is not itself providing control or service of L2 devices -- this is TSN's job.  The fact that DetNet is all about behavior of L3 nodes (i.e., IP and PW/MPLS) and not L2 nodes (i.e., TSN bridges) is something the architecture should make unambiguous.

Thanks,
Lou

Cheers,

Lou

Dear Bala'zs

Thank you your for your consideration of these points. I will just pick up a few that need some further thought:

    DetNet transit node

           transit node.

============

   combinations:

[Balázs Varga A] Agree, but these are examples. Statement is for HSR-PRP.
So the question is whether we should expand the set of examples, particularly
to more accessible ones.

============

      end    R +                   v | ^                + E end

      system   +                   v | ^                +   system

                                 Figure 1

===========

   document:

      layer.

      layer.

      during forwarding.

============

===========

Best Regards

Stewart