Subject: [Detnet] WG Last Call: draft-ietf-detnet-architecture-05

From: Lou Berger <[email-masked]>
Date: Fri, 11 May 2018 17:43:44 -0400

All,

This starts a two-week working group last call on
draft-ietf-detnet-architecture-05

The working group last call ends on May 25.
Please send your comments to the detnet mailing list.

Positive comments, e.g., "I've reviewed this document
and believe it is ready for publication", are welcome!
This is useful and important, even from authors.

Thank you,
DetNet Chairs

---

From: "Andrew G. Malis" <[email-masked]>
Date: Mon, 14 May 2018 08:11:48 -0400

IMHO, this draft is ready for publication.

Cheers,
Andy

On Fri, May 11, 2018 at 5:43 PM, Lou Berger <[email-masked]> wrote:

> _______________________________________________
> detnet mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/detnet
>

---

From: Stewart Bryant <[email-masked]>
Date: Wed, 23 May 2018 17:02:05 +0100

Some WG last call comments on draft-ietf-detnet-architecture

Best Regards

Stewart

=============

Abstract

    The capabilities can be managed by
    configuration, or by manual or automatic network management.

SB> I am not sure that network management (at least in the IETF is
SB> quite the right term. Most people think of what is described here
SB> as automatic network management as a routing protocol.

===============

1.  Introduction

    Deterministic Networking (DetNet) is a service that can be offered by
    a network to DetNet flows.  DetNet provides these flows extremely low
s/flows/flows with/
=========

    Many applications of interest to Deterministic Networking
SB> That is surely the wrong way round - the applications are
SB> interested in DN rather than DN being interested in the Apps?

==========

    source
            An end system capable of sourcing a DetNet flow.

SB> Shouldn't that be a DetNet source, since you can have
SB> both DetNet and non-DetNet sources in a DetNet enabled network.

===========

     DetNet transit node
            A node operating at the DetNet transport layer, that utilizes
            link layer and/or network layer switching across multiple
            links and/or sub-networks to provide paths for DetNet service
            layer functions.  Optionally provides congestion protection
            over those paths.  An MPLS LSR is an example of a DetNet
            transit node.
SB> In that example it would have to be a DetNet enable/aware LSR. An
SB> ordinary LSR would not know anything about DetNet.

=============

2.2.  IEEE 802 TSN to DetNet dictionary

    This section also serves as a dictionary for translating from the
    terms used by the IEEE 802 Time-Sensitive Networking (TSN) Task Group
    to those of the DetNet WG.

    Listener
            The IEEE 802 term for a destination of a DetNet flow.

    relay system
            The IEEE 802 term for a DetNet intermediate node.

    Stream
            The IEEE 802 term for a DetNet flow.

    Talker
            The IEEE 802 term for the source of a DetNet flow.

SB> Whilst I cannot comment on IEEE definitions, I am suprised
SB> that in our text a Listener is not called an 802 TSN Listener etc

=============

    The paths are typically (but not necessarily) explicit
    routes, so that they cannot suffer temporary interruptions caused by
    the reconvergence of routing or bridging protocols.

SB> I think cannot is too strong. I think the best that you can say
SB> is "do not normally".
SB>

============

    These three techniques can be applied independently, giving eight
    possible combinations, including none (no DetNet), although some
    combinations are of wider utility than others.  This separation keeps
    the protocol stack coherent and maximizes interoperability with
    existing and developing standards in this (IETF) and other Standards
    Development Organizations.  Some examples of typical expected
    combinations:

    o  Explicit routes plus service protection are exactly the techniques
       employed by [HSR-PRP].  Explicit routes are achieved by limiting
       the physical topology of the network, and the sequentialization,
       replication, and duplicate elimination are facilitated by packet
       tags added at the front or the end of Ethernet frames.

SB> ER can be done virtually as well as physically. RSVP is a type of
SB> ER, as is Adj-SID based SR, and we can design other types.

    o  Congestion protection alone is is offered by IEEE 802.1 Audio
       Video bridging [IEEE802.1BA-2011].  As long as the network suffers
       no failures, zero congestion loss can be achieved through the use
       of a reservation protocol (MSRP), shapers in every bridge, and a
       bit of network calculus.

SB> Network calculus needs a reference. It is not well known outside detnet.

================

3.2.3.  Jitter Reduction

    A core objective of DetNet is to enable the convergence of Non-IP
    networks onto a common network infrastructure.

SB> I think that  this needs to be clarified that it is the convergence
SB> of sensitive Non-IP. PW has been doing this convergence since inception,
SB> and can handle what some would expect to be sensitive applications
SB> such as TDM telephony quite well.

    This requires the
    accurate emulation of currently deployed mission-specific networks,
    which typically rely on point-to-point analog (e.g. 4-20mA
    modulation) and serial-digital cables (or buses) for highly reliable,
    synchronized and jitter-free communications.  While the latency of
    analog transmissions is basically the speed of light, legacy serial
    links are usually slow (in the order of Kbps) compared to, say, GigE,
    and some latency is usually acceptable.  What is not acceptable is
    the introduction of excessive jitter, which may, for instance, affect
    the stability of control systems.
SB> See above, I think this needs a few caveats.

===============

    Packet replication and elimination, also known as seamless redundancy
    [HSR-PRP], or 1+1 hitless protection, is a function of the DetNet
    service layer.  It involves three capabilities:

SB> I am concerned at the prominence of the HSR-PRP reference. As far
SB> as I can tell this is only available for fee of 300 Euro.

============

    In the simplest case, this amounts to replicating each packet in a
    source that has two interfaces, and conveying them through the
    network, along separate paths, to the similarly dual-homed
    destinations, that discard the extras.  This ensures that one path
    (with zero congestion loss) remains, even if some intermediate node
    fails.  The sequence numbers can also be used for loss detection and
    for re-ordering.

SB> Sure, but of course you need to do SRLG risk analysis, and even then
SB> two failures are not unknowm.

===========

                     Packet replication and elimination

                 > > > > > > > > > relay > > > > > > > >
                > /------------+ R node E +------------\ >
               > /                  v + ^               \ >
       end    R +                   v | ^                + E end
       system   +                   v | ^                +   system
               > \                  v + ^               / >
                > \------------+ R relay E +-----------/ >
                 > > > > > > > > >  node > > > > > > > >

                                  Figure 1

    Packet replication and elimination does not react to and correct
    failures; it is entirely passive.  Thus, intermittent failures,
SB> I think it copes with intermittent failures OK.

     mistakenly created packet filters, or misrouted data is handled just
    the same as the equipment failures that are detected handled by
    typical routing and bridging protocols.

================

    Transit nodes see DetNet nodes as end points.  All
    DetNet enabled nodes are connect to sub-networks, where a point-to-
SB> s/connect/connected/

===========

    o  DetNet t-aware: An extant TSN system.  It knows about some TSN
       functions (e.g., reservation), but not about replication/
       elimination.
SB> Generic TSN is OK, but I would be cautious of requiring IEEE TSN.

===========

  o  DetNet s-aware: An extant IEC 62439-3 system.  It supplies
       sequence numbers, but doesn't know about zero congestion loss.

SB> IEC62439 should surely be clarified as just an example.

===========

4.3.1.  DetNet flow types

    A DetNet flow can have different formats during while it is
SB> s/during while/as/

===========

    transported between the peer end systems.  Therefore, the following
    possible types / formats of a DetNet flow are distinguished in this
    document:

    o  App-flow: native format of a DetNet flow.  It does not contain any
       DetNet related attributes.

    o  DetNet-t-flow: specific format of a DetNet flow.  Only requires
       the congestion / latency features provided by the Detnet transport
       layer.

    o  DetNet-s-flow: specific format of a DetNet flow.  Only requires
       the replication/elimination feature ensured by the DetNet service
       layer.

    o  DetNet-st-flow: specific format of a DetNet flow.  It requires
       both DetNet service layer and DetNet transport layer functions
       during forwarding.
SB> I find the relisting of these types confusing. Wheren't they defined
SB> in the section above?

===========

4.3.2.  Source guarantees

    For the purposes of congestion protection, DetNet flows can be
    synchronous or asynchronous.  In synchronous DetNet flows, at least
    the intermediate nodes (and possibly the end systems) are closely
    time synchronized, typically to better than 1 microsecond.

SB> Are you talking about the flows being sync, or the network handling 
of the
SB> the flows here. We have carried synchronous traffic over MPLS in the 
past
SB> so I think this is a topic that requires quite careful handling to 
make sure that
SB> we do not set up an over-engineered solution.

==========

    The source promises that these limits will not be exceeded.  If the
    source transmits less data than this limit allows, the unused
    resources such as link bandwidth can be made available by the system
SB> s/can be made/to be made/

==========

4.4.  Traffic Engineering for DetNet

    Traffic Engineering Architecture and Signaling (TEAS) [TEAS] defines
    traffic-engineering architectures for generic applicability across
    packet and non-packet networks.  From TEAS perspective, Traffic
SB> s/from/from a/

==========

    ..... to translate from a flow
    specification to a set of values for the managed objects that control
    each relay or end system.  The IEEE 802 has specified (and is
SB> Should that be IEEE 802 Standards Group or some such?

=============

    The DetNet network reference model is shown in Figure 8 for a DetNet-
    Service scenario (i.e. between two DetNet-UNIs).  In this figure, the
    end systems ("A" and "B") are connected directly to the edge nodes of
    the IP/MPLS network ("PE1" and "PE2").

SB> I am confused as to when we should use the terms "relay node", "edge 
node"
SB> and "PE". There seem.s to be inconsistency in the project on the use
SB> of these terms

===========

    The tunnel between the service instances may have some special
    characteristics.  For example, in case of a "packet PW" based tunnel,

SB> I don't think we should use the term "packet PW" since
SB> that points to a specific PW technology and that is not
SB> the direction we are going in the data-plane part of the
SB> project.
SB> I think the text above and the text in the rest of the
SB> para is confusing for the reader.
SB> I suggest that we write something much more generic and
SB> let the DP team have a clear hand to design a good solution.

    there are differences in the usage of the packet PW for DetNet
    traffic compared to the network model described in [RFC6658]. In the
    DetNet scenario, the packet PW is used exclusively by the DetNet
    flow, whereas [RFC6658] states: "The packet PW appears as a single
    point-to-point link to the client layer.  Network-layer adjacency
    formation and maintenance between the client equipments will follow
    the normal practice needed to support the required relationship in
    the client layer ... This packet pseudowire is used to transport all
    of the required layer 2 and layer 3 protocols between LSR1 and LSR2".

===========

    The need for a lower-level DetNet node to be aware of individual
    higher-layer flows is not unique to DetNet.  But, given the endless
    complexity of layering and relayering over tunnels that is available
    to network designers, DetNet needs to provide a model for flow
    identification that is at least somewhat better than packet
SB> "at least somewhat better is a strange turn of phrase"

===========

    Thus, rather than packet inspection, there is the option to export
    higher-layer information to the lower layer.  The requirement to
    support one or the other method for flow identification (or both) is
    the essential complexity that DetNet brings to existing control plane
    models.

SB> A very strange way of expressing the point, sounds like complexity
SB> is a virtue.

===========

Flow-IDs

    The additional (domain specific) Flow-ID can be

    o  created by a domain specific function or

    o  derived from the Flow-ID added to the App-flow,

    so that it must be unique inside the given domain.
SB> very strange English.

    Note that the
    Flow-ID added to the App-flow is still present in the packet, but
    transport nodes may lack the function to recognize it; that's why the
    additional Flow-ID is added (pushed).

SB> Again rather strange English and an unnecessary pointer to a particular
SB> implementation technique (pushed). Same comment applies to para 1 of 
4.7.3

============

4.9.  Provisioning model

SB> Sections 4.9.1 and 4.9.2 seem rather tentative for an RFC
SB> that we are about to publish.

4.9.1.  Centralized Path Computation and Installation

4.9.2.  Distributed Path Setup

==============

5.  Security Considerations

    Furthermore, in a control system where millions of dollars of
    equipment, or even human lives, can be lost if the DetNet QoS is not
    delivered, one must consider not only simple equipment failures,
    where the box or wire instantly becomes perfectly silent, but bizarre
    errors such as can be caused by software failures.

SB> Bizare is a strange term to use. I am sure that later reviewers
SB> will ask for more text on this.
SB> I might use the term "subtle, complex, intermittent and state dependent"

============

    [HSR-PRP]  IEC, "High availability seamless redundancy (HSR) is a
               further development of the PRP approach, although HSR
               functions primarily as a protocol for creating media
               redundancy while PRP, as described in the previous
               section, creates network redundancy.  PRP and HSR are both
               described in the IEC 62439 3 standard.",
               <http://webstore.iec.ch/webstore/webstore.nsf/
               artnum/046615!opendocument>.

SB> Not available at the time of this review, but my recollection is that
SB> this is not readily available without paying a large fee.

===========

    [ISA95]    ANSI/ISA, "Enterprise-Control System Integration Part 1:
               Models and Terminology", 2000,
               <https://www.isa.org/isa95/>.

SB> Should that be 2000, or 2010.
SB> Note that this seems to be a very expensive document to access.

=========

---

From: "Black, David" <[email-masked]>
Date: Fri, 25 May 2018 21:26:28 +0000

Looking at this from a Transport (Area) viewpoint, I have a few mostly editorial comments:

-- Section 3.1

   o  Minimum and maximum end-to-end latency from source to destination;
      timely delivery and jitter avoidance derive from these constraints

"jitter avoidance" -> "limits on jitter"
as jitter cannot be entirely avoided.

   o  Probability of loss of a packet, under various assumptions as to
      the operational states of the nodes and links.  A derived property
      is whether it is acceptable to deliver a duplicate packet, which
      is an inherent risk in highly reliable and/or broadcast transmissions

"derived" does not seem to be the right word in "derived property" and
the whole second sentence seems off.   Suggest:

   o  Probability of loss of a packet, under various assumptions as to
      the operational states of the nodes and links.  If packet replication
      is used to reduce the probability of packet loss, then a related
      property is the probability (may be zero) of delivery of a duplicate
      packet.  Duplicate packet delivery
      is an inherent risk in highly reliable and/or broadcast transmissions.

-- Section 3.2

Text in 3.1 lists three techniques used by DetNet to provide QoS, but
section 3.2 (Mechanisms to achieve DetNet QoS) has five subsections.
An introductory paragraph in 3.2 (before 3.2.1) to explain how the five
mechanisms relate to the three techniques would be helpful.  As 3.2.3 on
jitter reduction is related to 3.2.1 on congestion protection, it might
help to swap the order of 3.2.2 and 3.2.3 so that the congestion
protection material is contiguous.

-- Section 4.1.1, after Figure 2

   The layers are, from top to bottom:

Not exactly - the list that follows does not contain an entry per layer.  Suggest

   The functionality shown in Figure 2 is:

and I would move OAM out of the list to its own paragraph at the end of
Section 4.1.1, as OAM is not shown in Figure 2.

-- Section 4.3.2 Source guarantees

Section title seems off, as nothing is being guaranteed to the source.
Would "Source transmission behavior" or Source transmission requirements"
be clearer?

   The source promises that these limits will not be exceeded.

I don't understand "promises" - who/what is the recipient of the promise, and 
how is the promise made?  Suggest:

   The source is required not to exceed these limits in order to obtain DetNet service.

-- Section 4.3.3 Incomplete Networks

   as
   extra buffering, and thus extra latency, must be allocated at points
   downstream from the non-DetNet intermediate node for a DetNet flow.

I don't think "extra latency" is being "allocated" - in addition, jitter should be
mentioned.   Suggest:

   as
   extra buffering must be allocated at points
   downstream from the non-DetNet intermediate node for a DetNet flow.
   This extra buffering may increase latency and/or jitter.

-- Section 4.10.  Scaling to larger networks

   The DetNet data plane, in
   order to support larger numbers of DetNet flows, must support the
   aggregation of DetNet flows into tunnels, , which themselves can be
   viewed by the transit nodes' data planes largely as individual DetNet
   flows.

That sentence makes two points:
  a) Flow aggregation is required for scaling.
  b) That flow aggregation has to employ tunnels.

The first point is supported by the text in this section, but the second point
does not appear to have supporting text.  It appears that tunnels are necessary
because: 
	- Each aggregated flow has to be a DetNet flow, and hence has to have
		its own 	DetNet flow ID.
	- The DetNet flow IDs of the flows that comprise an aggregated
		flow have to be carried through the aggregated flow in
		order to correctly demux to those original DetNet flows
		at the downstream end of flow aggregation.
A sentence or two should be added to explain this.

Thanks, --David
p.s.  I see that I'm already listed in the Acknowledgements section - that is appreciated.

> -----Original Message-----
> From: detnet [mailto:[email-masked]] On Behalf Of Lou Berger
> Sent: Friday, May 11, 2018 5:44 PM
> To: DetNet WG
> Cc: DetNet Chairs
> Subject: [Detnet] WG Last Call: draft-ietf-detnet-architecture-05
> 
> All,
>

---

From: Lou Berger <[email-masked]>
Date: Wed, 30 May 2018 14:30:44 -0400

Hello,

     The WG last call on this document is closed.  Authors, please 
address any comments received on or off list -- and let the WG know how 
each issues is resolved, as well as when you think the document is ready 
to progress. (On list issues should be responded to directly.)

Thank you,

Lou

PS I also have comments as part of my shepherd review and will send them 
to the list shorty.