Subject: [Detnet] consolidated response: IP Data Plane draft comments

From: Lou Berger <[email-masked]>
Date: Sun, 6 Oct 2019 16:47:28 -0400

David,

     My apologies on the time to get back to this topic.  Before the 
last IETF you raised a number of points on the IP data plane document.  
For context, I believe the threads are found in:
https://mailarchive.ietf.org/arch/msg/detnet/ykpI6gbH2L705BBjmHbnsZraxig
https://mailarchive.ietf.org/arch/msg/detnet/phi4OBWijlp7T4_HML2YvWfoaas
https://mailarchive.ietf.org/arch/msg/detnet/th2Yj5Z4_8VtBsEOF1UfLejMhpM
https://mailarchive.ietf.org/arch/msg/detnet/qvs-S0Nrm1B8w8JYO7zDLFwpTFQ

In re reviewing the discussions, I think we missed the ECN field. Based 
on this and a general reread, I have put together a proposed set of 
changes to the published draft.  The changes are  available at

https://github.com/detnet-wg/data-plane-drafts/commit/d3a733556804f11abbedbd128e3c62786418ddc8

(text format can be received at 
https://xml2rfc.tools.ietf.org/cgi-bin/xml2rfc.cgi?url=https://raw.githubusercontent.com/detnet-wg/data-plane-drafts/working/lb/ip-tos-update/ip/draft-ietf-detnet-ip.xml)

the following changes are proposed:
OLD183,195c185,202
<    The DetNet IP data plane uses "6-tuple" based flow identification,
<    where 6-tuple refers to information carried in IP and higher layer
<    protocol headers.  The 6-tuple referred to in this document is the
<    same as that defined in [RFC3290].  Specifically 6-tuple is
<    (destination address, source address, IP protocol, source port,
<    destination port, and differentiated services (DiffServ) code point
<    (DSCP).  General background on the use of IP headers, and 5-tuples,
<    to identify flows and support Quality of Service (QoS) can be found
<    in [RFC3670].  [RFC7657] also provides useful background on the
<    delivery of DiffServ and "tuple" based flow identification.
<    Referring to a 6-tuple allows DetNet nodes to forward packets with
<    the 6-tuple as is or remap the DSCP where required by the DetNet
<    service.
NEW
 >    The DetNet IP data plane primarily uses "6-tuple" based flow
 >    identification, where 6-tuple refers to information carried in IP and
 >    higher layer protocol headers.  The 6-tuple referred to in this
 >    document is the same as that defined in [RFC3290]. Specifically
 >    6-tuple is (destination address, source address, IP protocol, source
 >    port, destination port, and differentiated services (DiffServ) code
 >    point (DSCP).  General background on the use of IP headers, and
 >    5-tuples, to identify flows and support Quality of Service (QoS) can
 >    be found in [RFC3670].  [RFC7657] also provides useful background on
 >    the delivery of DiffServ and "tuple" based flow identification.
 >
 >    The DetNet IP data plane also allows for optional matching on two
 >    additional data fields.  The optional fields are the ECN Field, as in
 >    [RFC3168], and the IPv6 flow label field, as defined in [RFC8200].
 >
 >    Generally the fields used in flow identification are forward
 >    unmodified but modification is allowed, for example to a DSCP value,
 >    when required by the DetNet service.

OLD
<    masks, prefixes and ranges.  IP tunnels may also be used to support
NEW
 >    masks, lists, prefixes and ranges.  IP tunnels may also be used to

OLD
437,438c444,446
<    of the 6-tuple i.e., the typical 5-tuple enhanced with the DSCP code,
<    uniquely identifies a DetNet service flow.
NEW
 >    of the 6-tuple i.e., the typical 5-tuple enhanced with the DSCP and
 >    previously mentioned two optional fields, uniquely identifies a
 >    DetNet service flow.

OLD
590,593c598,608
<    MUST support bitmask based matching, where bits set to one (1) in the
<    bitmask indicate which subset of the bits in the field are to be used
<    in determining a match.  Note that all bits set to zero (0) value as
<    a bitmask effectively means that these fields are ignored.
NEW
 >    MUST support list based matching of DSCP values, where the list is
 >    composed of possible field values that are to be considered when
 >    identifying a specific DetNet flow.  Implementations SHOULD allow for
 >    this field to be ignored for a specific DetNet flow.
 >
 >    Implementations of this document MUST allow the ECN field to be
 >    ignored as part of DetNet flow identification. Additionally,
 >    implementations SHOULD support identification of DetNet flows based
 >    on the value carried in the ECN field.  When this field is used to
 >    identify a specific DetNet flow, implementations MUST support a list
 >    of ECN values that match a specific slow.

OLD
705c720,729
<    o  IPv4 Type of Service and IPv6 Traffic Class Fields.
<
<    o  IPv4 Type of Service and IPv6 Traffic Class Field Bitmask, where a
<       zero (0) effectively means that theses fields are ignored.
NEW
 >    o  For the IPv4 Type of Service and IPv6 Traffic Class Fields:
 >
 >       *  If the DSCP field is to be used in flow identification.
 >          Ignoring the DSCP filed is optional.
 >
 >       *  When the DSCP field is used in flow identification, a list of
 >          field values that may be used by a specific flow.
 >
 >       *  If the ECN field is to be used in flow identification.
 >          Matching based on ECN filed values is optional.
 >
 >       *  When ECN field is used in flow identification, a list of field
 >          values that may be used by a specific flow.

Please let me know if this addresses all your comments in an acceptable 
fashion or if you'd like to discuss further.

(BTW I think this is the sole open set of issues on the document.)

Thank you!

Lou

---

From: "Black, David" <[email-masked]>
Date: Tue, 8 Oct 2019 14:02:39 +0000

Lou,

> fashion or if you'd like to discuss further.

The proposed text does not address the comments.

[A] ECN

The proposed text effectively ignores the comments, and fails to address the problem.

The ECN field MUST NOT be used for flow identification, period (e.g., see RFC 3290, which does not include ECN in the 6-tuple).  In particular, it would be wrong for two DetNet flows to differ solely on the values used in the ECN field.  If there's a disagreement, please cite text from RFC 3168 that justifies this proposed ECN field usage.

[B] 5-tuples and 6-tuples

As I noted in one of the referenced messages in response to a question:

> > Are you okay with dropping "While 5-tuples suffice to identify DetNet flows,"
> Not by itself - if that text is dropped, I'd want to add a couple of sentences
> (perhaps not in Section 3) to explain how DetNet flows differ from Diffserv flows,
> i.e., "microflows" in defined in RFC 2475 (Diffserv Architecture).

The heads up for everyone is that if multiple DetNet flows differ only in DSCP (i.e., IP addresses, transport protocol and ports are the same), then non-DetNet network management tools that deal with traffic flows will lump those flows together.  Is it at least possible to prohibit (or at least strongly discourage, i.e., "SHOULD NOT") use of the same 5-tuple for DetNet and non-DetNet traffic?  That would at least contain the resulting network management and operational effects to DetNet?

The use of DSCP for flow identification makes the OAM problem much harder for the IP data plane by dramatically increasing the difficulty of getting DetNet IP OAM traffic to fate-share with the DetNet traffic for which the OAM functionality is intended (it may be necessary to resort to shoving an MPLS label in there to identify OAM traffic and requiring nodes that originate or terminate DetNet OAM packets to understand enough of MPLS to deal with this).

Also, be aware that if two DetNet flows differ only in DSCP, most existing network transport protocols will have problems dealing with that situation - see RFC 7657 for details.

Thanks, --David

> -----Original Message-----
> From: Lou Berger <[email-masked]>
> Sent: Sunday, October 6, 2019 4:47 PM
> To: Black, David
> Cc: [email-masked]
> Subject: consolidated response: IP Data Plane draft comments
> 
> 
> [EXTERNAL EMAIL]
> 
> David,
> 
> https://github.com/detnet-wg/data-plane-
> drafts/commit/d3a733556804f11abbedbd128e3c62786418ddc8
> 
> https://xml2rfc.tools.ietf.org/cgi-
> bin/xml2rfc.cgi?url=https://raw.githubusercontent.com/detnet-wg/data-
> plane-drafts/working/lb/ip-tos-update/ip/draft-ietf-detnet-ip.xml)
>

---

From: "Black, David" <[email-masked]>
Date: Wed, 9 Oct 2019 13:30:58 +0000

If it helps, I'll be happy to join one of the Monday calls on the data plane drafts to discuss this further.

Thanks, --David

> From: Black, David <[email-masked]>
> Sent: Tuesday, October 8, 2019 10:03 AM
> To: Lou Berger
> Cc: [email-masked]; Black, David
> Subject: RE: consolidated response: IP Data Plane draft comments
> 
> Lou,
> 
> The ECN field MUST NOT be used for flow identification, period (e.g., see RFC
> 3290, which does not include ECN in the 6-tuple).  In particular, it would be
> wrong for two DetNet flows to differ solely on the values used in the ECN
> field.  If there's a disagreement, please cite text from RFC 3168 that justifies
> this proposed ECN field usage.
> 
> The heads up for everyone is that if multiple DetNet flows differ only in DSCP
> (i.e., IP addresses, transport protocol and ports are the same), then non-
> DetNet network management tools that deal with traffic flows will lump
> those flows together.  Is it at least possible to prohibit (or at least strongly
> discourage, i.e., "SHOULD NOT") use of the same 5-tuple for DetNet and
> non-DetNet traffic?  That would at least contain the resulting network
> management and operational effects to DetNet?
> 
> The use of DSCP for flow identification makes the OAM problem much
> harder for the IP data plane by dramatically increasing the difficulty of getting
> DetNet IP OAM traffic to fate-share with the DetNet traffic for which the
> OAM functionality is intended (it may be necessary to resort to shoving an
> MPLS label in there to identify OAM traffic and requiring nodes that originate
> or terminate DetNet OAM packets to understand enough of MPLS to deal
> with this).
> 
> Also, be aware that if two DetNet flows differ only in DSCP, most existing
> network transport protocols will have problems dealing with that situation -
> see RFC 7657 for details.
> 
> Thanks, --David
>

---

From: Lou Berger <[email-masked]>
Date: Wed, 9 Oct 2019 22:41:32 -0400

Hi David,

Thanks for the response, see below.

----------
On October 8, 2019 10:04:00 AM "Black, David" <[email-masked]> wrote:

> The ECN field MUST NOT be used for flow identification, period (e.g., see 
> RFC 3290, which does not include ECN in the 6-tuple).  In particular, it 
> would be wrong for two DetNet flows to differ solely on the values used in 
> the ECN field.  If there's a disagreement, please cite text from RFC 3168 
> that justifies this proposed ECN field usage.
>

I think there's some cross usage of the term flow here.   DetNet flow 
identification is the traffic classification function that enables 
DetNet to deliver different service, including paths through a network 
to different flows.  ECN markings already impact how packets are handled 
in ECN enabled routers (DROP for 00 for DE packets).  Also, I don't see 
anything in the RFCs that would preclude path selection based on 
different DSCPs or different ECN field values -- It is even covered 
under 'split paths' albeit from the perspective of a broken 
implementation.  As a possible example, I could see ECT and non-ect 
traffic routed over different paths and am not sure where this is 
precluded by the reference RFCs.

> The heads up for everyone is that if multiple DetNet flows differ only in 
> DSCP (i.e., IP addresses, transport protocol and ports are the same), then 
> non-DetNet network management tools that deal with traffic flows will lump 
> those flows together.  Is it at least possible to prohibit (or at least 
> strongly discourage, i.e., "SHOULD NOT") use of the same 5-tuple for DetNet 
> and non-DetNet traffic?  That would at least contain the resulting network 

I'm not sure how we can preclude what operators choose to do.  Vendors 
on the other hand...

How about adding:

It is important to note that the DSCP values used within a DetNet domain 
may not be meaningful outside of that DetNet domain as the values may 
have no relationship to standard PHB and DSCP values.  It is REQUIRED 
that routers that operate in both DetNet and non-DetNet domains support 
remapping or setting the DSCP value of any flow originating in a DetNet 
domain.

>
> The use of DSCP for flow identification makes the OAM problem much harder 
> for the IP data plane by dramatically increasing the difficulty of getting 
> MPLS label in there to identify OAM traffic and requiring nodes that 
> originate or terminate DetNet OAM packets to understand enough of MPLS to 
> deal with this).
For the MPLS data plane, I suspect this will be required.  For IP, given 
the complexities of ECMP, I don't see how this behavior notably 
complicates OAM, but also agree it will need to be considered.

> network transport protocols will have problems dealing with that situation 
> - see RFC 7657 for details.

sure.  but then it should be using different DSCP values.

Lou

PS WRT your offer to join one of the data plane calls, that would be great!

> _______________________________________________
> detnet mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/detnet

---

From: "Black, David" <[email-masked]>
Date: Thu, 10 Oct 2019 15:10:44 +0000

Lou,

> in ECN enabled routers (DROP for 00 for DE packets).  

I'm drawing a bright-line distinction between different service in network nodes (both DSCP and ECN are used for this purpose) and selection of paths.

In other words, if "including paths through the network" were removed from the above text snippet, I would strongly agree.

> Also, I don't see
> different DSCPs or different ECN field values

I firmly disagree on those two:
DSCP:- Selection of different paths based on DSCP is problematic for network operations and management tools.
ECN: I believe that different path selection based on ECN values is effectively prohibited by RFC 3168, and is completely inconsistent with prior and ongoing Transport Area work.

To summarize what I think ought to be done:
- Path selection based on DSCP ought to be a "SHOULD NOT" requirement.  RFC 7657 is relevant to this.
- Path selection based on ECN has to be a "MUST NOT" requirement - see RFC 3168.

> As a possible example, I could see ECT and non-ect

Please read RFC 3168 to better understand why that is a seriously bad idea.

> > Is it at least possible to prohibit (or at least

I'm not looking for "preclude" which would be expressed as "MUST NOT."  What I'm looking for is a strong "here be dragons" "SHOULD NOT" warning about mixing DetNet and non-DetNet traffic on the same 5-tuple (e.g., that sure looks like an opportunity to misdirect DetNet OAM traffic into non-DetNet paths and forwarding planes).  Then operators who choose to do otherwise have only themselves to credit for the consequences.

> sure.  but then it should be using different DSCP values.

I'm not sure what "it" refers to, but if that was meant to imply that existing network transport protocols are not or never appropriate for use with DetNet, then I would strongly disagree.

The rest of the context is that in end systems, DSCP is not used to choose the "socket" or equivalent - traffic is muxed/demuxed to/from transport protocol sessions/instances based on 5-tuple (not including DSCP) with DSCPs being used to label flows, but not to subdivide any 5-tuple flow into multiple 6-tuple sub-flows.   If splitting a 5-tuple flow into multiple sub-flows is what's wanted, then the WebRTC example ought to be followed where header info above the IP header (and above the UDP header) is used to do the mux/demux so that multiple WebRTC sub-flows can use the same 5-tuple.

Thanks, --David

> From: detnet <[email-masked]> On Behalf Of Lou Berger
> Sent: Wednesday, October 9, 2019 10:42 PM
> To: Black, David
> Cc: [email-masked]
> Subject: Re: [Detnet] consolidated response: IP Data Plane draft comments
> 
> 
> [EXTERNAL EMAIL]
> 
> Hi David,
> 
> >> (perhaps not in Section 3) to explain how DetNet flows differ from
> Diffserv
> >> flows,
> harder

---

From: Lou Berger <[email-masked]>
Date: Fri, 11 Oct 2019 06:59:55 -0400

David,

----------
On October 10, 2019 11:13:54 AM "Black, David" <[email-masked]> wrote:

> I'm drawing a bright-line distinction between different service in network 
> nodes (both DSCP and ECN are used for this purpose) and selection of paths.
>

And I believe we are trying to ensure that the that the DetNet data plane 
can be supported on the widest variety of existing and future hardware. 
Which to me means not introducing any restrictions that don't exist today 
or overly specify behaviors not required by detnet.

> In other words, if "including paths through the network" were removed from 
> the above text snippet, I would strongly agree.
>
>> Also, I don't see

I don't disagree, but we don't prevent  operators to 'play with scissors'. 
Sometimes such flexibility even results in new capabilities.

Adding text that points this out seems completely reasonable.

> ECN: I believe that different path selection based on ECN values is 
> effectively prohibited by RFC 3168, and is completely inconsistent with 
> prior and ongoing Transport Area work.
>

Where is this prohibition stated? Referencing existing specification 
certainly makes sense.

Although, if a system is smart enough to route ecn enabled packets over ecn 
supporting routers I'm not sure why this would be a bad thing. But if the 
restriction already exists, it exists...

> Please read RFC 3168 to better understand why that is a seriously bad idea.

Perhaps this is a good discussion point for Monday's call.

> I'm not looking for "preclude" which would be expressed as "MUST NOT."  
> What I'm looking for is a strong "here be dragons" "SHOULD NOT" warning 
> about mixing DetNet and non-DetNet traffic on the same 5-tuple (e.g., that 
> sure looks like an opportunity to misdirect DetNet OAM traffic into 
> non-DetNet paths and forwarding planes).  Then operators who choose to do 
> otherwise have only themselves to credit for the consequences.
>

Okay. I'll work on some language for this and send it to the list before 
Monday's call.

> I'm not sure what "it" refers to, but if that was meant to imply that 
> existing network transport protocols are not or never appropriate for use 
> with DetNet, then I would strongly disagree.
>
> The rest of the context is that in end systems, DSCP is not used to choose 
> the "socket" or equivalent - traffic is muxed/demuxed to/from transport 
> protocol sessions/instances based on 5-tuple (not including DSCP) with 
> DSCPs being used to label flows, but not to subdivide any 5-tuple flow into 
> multiple 6-tuple sub-flows.   If splitting a 5-tuple flow into multiple 
> sub-flows is what's wanted, then the WebRTC example ought to be followed 
> where header info above the IP header (and above the UDP header) is used to 
> do the mux/demux so that multiple WebRTC sub-flows can use the same 5-tuple.
>

I hear you and agree you certainly wouldn't expect this to happen for a 
single 5-tuple.  This said,  I have seen some real networks, and suspect 
there are more, that route traffic differently based on their dscps - and 
this is nothing more than we are allowing for in detnet.

I look forward to talking to you on Monday

Lou

---

From: "Black, David" <[email-masked]>
Date: Fri, 11 Oct 2019 20:26:51 +0000

Lou,

> or overly specify behaviors not required by detnet.

In my view, that also means not departing significantly from the network architecture on which that hardware is based.  The underlying network architecture aspect that matters here is that both DSCP and ECN are intended to provide multiple markings within individual flows, not subdivide flows into sub-flows.  In more detail ...

-- DSCP --

The proposal to use DSCP for path selection is certainly "creative" :-), as Diffserv is primarily concerned with forwarding behaviors, see RFC 2475 (An Architecture for Differentiated Services).  That said, in light of the warnings in RFC 7657 (Differentiated Services (Diffserv) and Real-Time Communication) about use of multiple DSCPs, suitable "there be dragons here" warnings, i.e., "SHOULD NOT" language with explanations of what can go wrong, ought to likewise suffice for DetNet.

An important RFC 7657 warning (in Section 6) to keep in mind is:

   o  Cannot rely on end-to-end preservation of DSCPs as network node
      remarking can change DSCPs and remove drop precedence distinctions
      (see Section 3.2).  For example, if a source uses drop precedence
      distinctions within an AF class to identify different types of
      video frames, using those DSCP values at the receiver to identify
      frame type is inherently unreliable.

There is a lot of deployed network equipment that remarks DSCPs, so relying on DSCPs to identify DetNet flows (or even DetNet traffic), and especially to distinguish DetNet traffic from non-DetNet traffic, has potentially problematic hardware interactions.  In contrast, if any 5-tuple is either all-DetNet or all-not-DetNet, network DSCP remarking ought not to result in moving DetNet traffic into a non-DetNet data plane at network nodes or moving non-DetNet traffic into a DetNet data plane, both of which look like robustness benefits for DetNet.  There is also an analogous robustness benefit for DetNet network edge filtering that prevents DetNet IP traffic from escaping into non-DetNet networks - if the filters are 5-tuple-based, DSCP remarking ought not to result in DetNet traffic bypassing an edge filter that is intended to block Detnet traffic.

Overall, this DSCP topic still feels like it merits "SHOULD/SHOULD NOT" requirements, not "MUST/MUST NOT" requirements (reminder: what I'm looking for is roughly - SHOULD NOT mix DetNet and non-DetNet traffic within a single 5-tuple).

OTOH ...

-- ECN --

> restriction already exists, it exists...

It would definitely be a bad thing if it is based on the value of the ECN field in the IP header.  One problem arises from the requirement (in Section 6.1.5 of RFC 3168) that when a TCP session uses ECN, all retransmitted packets have to be marked with Not-ECT, i.e., as not using ECN.   As a result, sending ECN and non-ECN traffic on different network paths invites reordering of packets in a TCP session, which is known to be problematic, e.g., if retransmitted packets are subject to more delay than non-retransmitted packets.  There's much more where this explanation came from (please do read RFC 3168).  The upshot is that DetNet needs to prohibit use of ECN for path selection, i.e., "MUST NOT" language that cites RFC 3168.

For completeness, I should not that there is experimental work in progress to support ECN for retransmitted packets, e.g., see RFC 8311 (Relaxing Restrictions on Explicit Congestion Notification (ECN) Experimentation), but there is also plenty of "running code" that does not use ECN for retransmitted packets in traffic that otherwise uses ECN.

Thanks, --David (co-author of RFCs 2475, 3168, 7657 & 8311)

> Sent: Friday, October 11, 2019 7:00 AM
> To: Black, David
> Cc: [email-masked]
> Subject: RE: [Detnet] consolidated response: IP Data Plane draft comments
> 
> 
> [EXTERNAL EMAIL]
> 
> David,
> 
> 
> ----------
> > - Path selection based on DSCP ought to be a "SHOULD NOT" requirement.
> RFC
> > 7657 is relevant to this.
> 
> 
> > - Path selection based on ECN has to be a "MUST NOT" requirement - see
> RFC
> > 3168.
> >
> >> > The proposed text effectively ignores the comments, and fails to
> address
> >> > the problem.
> >> >
> >> >> > Are you okay with dropping "While 5-tuples suffice to identify
> DetNet
> >> flows,"

---

From: Lou Berger <[email-masked]>
Date: Sat, 12 Oct 2019 18:20:16 -0400

Hi David,

> In my view, that also means not departing significantly from the network architecture on which that hardware is based.  The underlying network architecture aspect that matters here is that both DSC…

...

> Overall, this DSCP topic still feels like it merits "SHOULD/SHOULD NOT" requirements, not "MUST/MUST NOT" requirements (reminder: what I'm looking for is roughly - SHOULD NOT mix DetNet and non-Det…

Okay, how about adding the following to the end of section 4.1:

           DetNet aware applications and end systems SHOULD NOT mix
           DetNet and non-DetNet traffic within a single 5-tuple.

> It would definitely be a bad thing if it is based on the value of the ECN field in the IP header.  One problem arises from the requirement (in Section 6.1.5 of RFC 3168) that when a TCP session uses ECN, all retransmitted packets have to be marked with Not-ECT, i.e., as not using ECN.   As a result, sending ECN and non-ECN traffic on different network paths invites reordering of packets in a TCP session, which is known to be problematic, e.g., if retransmitted packets are subject to more delay than non-retransmitted packets.

This is certainly problematic!

> There's much more where this explanation came from (please do read RFC 3168).  The upshot is that DetNet needs to prohibit use of ECN for path selection, i.e., "MUST NOT" language that cites RFC 3168.

My reservation in putting in this restriction in the data plane is that 
I think this will be the first/only RFC document to have this 
restriction on the IP forwarding layer.  Furthermore I think what you're 
really saying is not about traffic classification but rather path 
selection.  It is a *very* valid point and I think is worth adding to 
the document even though  path selection is out of scope.   What do you 
think about adding a new forwarding sublayer section along the lines of:

4.3.3.  Path Selection

    While path selection algorithms and mechanisms are out of scope of
    the DetNet data plane definition, it is important to highlight the
    implications of DetNet IP flow identification on path selection and
    next hops.  As mentioned above, the DetNet IP data plane identifies
    flows using "6-tuple" header information as well as two additional
    optional header fields.  DetNet generally allows for both flow-
    specific traffic treatment and flow-specific next-hops.

    In non-DetNet IP forwarding, it is generally assumed that the same
    series of next hops, i.e., the same path, will be used for a
    particular 5-tuple or, in some cases, e.g., [RFC5120], for a
    particular 6-tuple.  This assumption is reflected in higher protocol
    levels, e.g., see [RFC3168].  Using different next hops for different
    5-tuples does not take any special consideration within a DetNet
    domain.  Care should be taken when using different next hops for the
    same 5-tuple.  Understanding of the application and transport
    protocol impact of using different next hops for the same 6-tuple is
    required.  Again, this impacts path selection for DetNet flows and
    this document only indirectly.

> For completeness, I should not that there is experimental work in progress to support ECN for retransmitted packets, e.g., see RFC 8311 (Relaxing Restrictions on Explicit Congestion Notification (E…

Allowing future experimentation/evolution of usage of the detnet 
forwarding plane is exactly the motivation for not restricting detnet 
more than general IP.

Thanks and talk to you on Monday.

Lou

PS Diff is on github: 
https://github.com/detnet-wg/data-plane-drafts/commit/ee5d5f71f4e52d402223e4ad4dd88e6c93465f64