Subject: [Detnet] Some thoughts on Replication and Elimination

From: Stewart Bryant <[email-masked]>
Date: Sat, 11 Nov 2017 23:09:54 +0000

I had some thoughts on how to look at replication and elimination.

I have written them up in draft format, although whether I upload the 
draft or not is to be determined. However I though that there was more 
to the issue than a simple email would easily handle.

However in in view of the uploading embago, and considering the 
shortness of the draft, I am posting it to the list so that it can be 
discussed.

- Stewart

---

From: Toerless Eckert <[email-masked]>
Date: Mon, 13 Nov 2017 00:28:28 +0100

Hi Steart

Thanks for this draft. I am quite interest to help co-author this further
if there is interest by the WG in this type of work. I would very much
hope there is, even though i would find this work most beneficial if
it was targetted as a normative reference describing this forwarding
model in a way that would allow it to be referenced by more IETF
solutions than "just" DetNet (not to diminish DetNet any way ;-).

The one example use case that is IMHO like DetNet missing a solid definition
how to do this is exactly is the often as "live-life" described dual forwarding
of IP multicast traffic in network devices:

IMHO, IP multicast can actually help to bring down the price in products
to support good live-live forwarding functions because in IP multicast
this method would have applicability to more traffic than in unicast:

In Unicast (e.g.: DetNet), the network cost is simply factor 2 in bandwidth
(high level, there are arguments to bring down that perceived cost).
In IP multicast it actually a cost-saving: Consider you loose a packet
that would be replicated to 10,000 users. And you need to protect againsts
that loss. All other approaches such as other forms of FEC or ARQ (retransmission)
have a relatively higher cost of bandwidth and operations than live-live.

Even though there are various vendor options to support live-live for multicast,
to the best of my knowledge, there are no implementations where the
replication/elimination (often called "merge" or "de-duplication") is performed
by sequence numbers in network devices. But i have not tracked this for
a few years across vendors. It has been demonstrated to be IMHO easily
possible in HW forwarding planes with sufficiently intelligent HW forwarding
planes as early as maybe 8..10 years back and should be easily be implementable
in e.g.: P4 as an example. The only instance of IETF work related to
this is IMHO RFC7431, but it does not include the actual sequence number
based duplication/de-duplication but only resorts to deplication based
on state based switchover. Something which is a common technique in netork
such a pseudowire redundancy and others if i remember correctly.

Doing sequence number based live-live has of course been the standard
mechanism how our retirement savings are gambled with by the financial
industry - since at least hmm.. 1997 i think: market data networks. No
single packet must be lost in those networks and IP multicast solutions
with live-live provided that now for 20 years.

Therefore it is also long time well known how to enginer SP networks
to support the dual-path transmission. But to the best of my knowledge,
they leave the duplication/de-duplication all to the application side.
The same is true for linear TV video, where SMPTE receiver functions do 
support this sequence number de-duplication as a standard function.
And just leave the dual path transmission to network equipment.  One
transport layer definition for this is RFC7198.

So, IMHO we are missing a simple, but solid definition of a network
devices forwarding plane functionality to duplicate with sequence
number insertion (or this may come from app side) and de-duplication
based on sequence number. The standard behavior of those forwarding
functions is IMHO best defined based on the visible parameter
of its behavior at the operator side (e.g.: Yang model) because
for any such high-value service you need to have some operational
parameter such as measured partially-lost/fully-lost/late etc. packets,
and a definition how to measure those pretty much requires a modelling
of the underlying duplicate elimination model. Maybe we could even
define this standard behavior via a reference P4 program. That could
be fun ;-)

To make it widely applicable, the specification
should really only refer to application of this forwarding plane function
to packet-flows identified by other means (aka: any other form of
looking up/identifying packet flows) and sequence numbers at
a defineable offset into packets. Therefore staying agnostic to
any specific choosen encap to carry the sequence numer. L2TPv3, RTP,
any form of other L2/MPS/etc. encap i don't know - does not matter.

Of course we would need to have in the same or another draft at least
one reference encap option that would allow to deploy a fully standardized
version of this (MPLS is fine given where SPs stand today ;-)).

Wrt to some of the statements in your document:

- If the replication is also inserting sequence numbers it would also
  be a stateful operation.

- duplicate transmission in itself is not a mean to counter congestion
  or make rate adaptation simpller (where rate adaptation is the IETF
  TSV requirement for reaction to congestion).  See also RFC7198.
  Doing rate-adaptive traffic in live-live environments would be
  a cool reaearch topic IMHO, but not something DetNet or the
  multicast use cases would immediatel profit from.

  Aka: The right formulation would be that live-live can help
  in a variety of loss situations:

  - total path loss on one side (link, node, linecard, ... failure).

  - BER type issues: loss of packets due to physcial tranmission
    errors - not congestion based (very common in copper, but also WiFi).

  - burst accumulation into queues. I think it
    is this case you where thinking of, but i think thats not
    called (ongoing) congestion, aka: it does not require
    downspeeding/rate adaptation. But live in queue/buffer
    management becomes a lot easier if you have some headroom
    here through live-live.

- I am not sure to what degree a definition would need to get into
  specific platform issues such as where/how to do these functions
  in a distributed fashion, e.g.: on ingres or egres and the like.
  As an example: BIER is also a very novel forwarding plane paradigm
  and all the initial platforms with asic acceleration we discussed
  BIER with where having quite some challenges figuring out how
  to do it, but that did not lead to any request to discuss/document
  any such device internals on the IETF drafts (now to become RFC).

  Aka: with that recent experience in mind i would stick to the
  external observable behavior, well define it (including above
  described operator interface/diagnostics as a key component),
  and then see what feedback we get.

Cheers
    Toerless

---

From: Norman Finn <[email-masked]>
Date: Mon, 13 Nov 2017 00:49:33 +0000

Toerless,

Who says that DetNet is unicast only??  The drafts haven't addressed this very well, yet.  But TSN has been multicast from Day 1.  The DetNet data plane will handle multicast.  The trick is to realize, as TSN has, that packet replication for reliability is *not* a multicast operation.  I say it's not multicast, because the branches will criss-cross, and that's something that multicast doesn't do.  The multicast trees spread out *after* the packet replication.  Those multicast paths are true trees, and the two (or more) multicast trees take different routes to the same places.

-- Norm
________________________________________
From: detnet [[email-masked]] on behalf of Toerless Eckert [[email-masked]]
Sent: Sunday, November 12, 2017 3:28 PM
To: [email-masked]; [email-masked]
Subject: Re: [Detnet] Some thoughts on Replication and Elimination

Hi Steart

Cheers
    Toerless

_______________________________________________
detnet mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/detnet

---

From: Toerless Eckert <[email-masked]>
Date: Mon, 13 Nov 2017 02:19:58 +0100

Hi Norm,

thanks for the note.

Yepp, of course, I should have qualified my text not
making it look as if i thought Detnet=unicast but with detnet even if unicast.

In other words: in pre-detnet IP solutions, we discussed live-live for unicast but my
impression was that it was never crucial enough to get done. 
With detnet the service criteria are so strict that that IMHO changes even for unicast.

Cheers
   Toerless

-- 
---
[email-masked]

---

From: Stewart Bryant <[email-masked]>
Date: Mon, 13 Nov 2017 02:09:24 +0000

From  the perspective of this document, the point to note is that the 
replicator (whether replicating multicast or unicast traffic) uses the 
same packet forwarder technology as multicast. This technology is 
ubiquidous and stateless.

It is the eliminator that is statefull and far more challanging.

- Stewart

---

From: Toerless Eckert <[email-masked]>
Date: Mon, 13 Nov 2017 03:18:03 +0100

In that case you're assuming some pre-existing sequence number generation mechanism. 

Its not sure to me whether thats a valid assumption or whether creation
of secquence number for specifically identified classes of packets and attaching the
sequence number to the packet is not also a feature often considered novel especially
in accelerated HW forwarding plane.

Cheers
    Toerless

On Mon, Nov 13, 2017 at 02:09:24AM +0000, Stewart Bryant wrote:
> 
> From  the perspective of this document, the point to note is that
> the replicator (whether replicating multicast or unicast traffic)
> uses the same packet forwarder technology as multicast. This
> technology is ubiquidous and stateless.
> 

-- 
---
[email-masked]

---

From: Balázs Varga A <[email-masked]>
Date: Mon, 13 Nov 2017 02:39:10 +0000

Hi,
Sequence numbers are generated: 
- by the source
- at the ingress UNI (if S/N not generated by the source or it cannot be (re-)used by the ingress PE) 
Cheers
Bala'zs

-----Original Message-----
From: detnet [mailto:[email-masked]] On Behalf Of Toerless Eckert
Sent: 2017. november 13. 10:18
To: Stewart Bryant <[email-masked]>
Cc: Norman Finn <[email-masked]>; [email-masked]
Subject: Re: [Detnet] Some thoughts on Replication and Elimination

Its not sure to me whether thats a valid assumption or whether creation of secquence number for specifically identified classes of packets and attaching the sequence number to the packet is not also a feature often considered novel especially in accelerated HW forwarding plane.

Cheers
    Toerless

On Mon, Nov 13, 2017 at 02:09:24AM +0000, Stewart Bryant wrote:
> >Thanks for this draft. I am quite interest to help co-author this 
> >further if there is interest by the WG in this type of work. I would 
> >very much hope there is, even though i would find this work most 
> >beneficial if it was targetted as a normative reference describing 
> >this forwarding model in a way that would allow it to be referenced 
> >by more IETF solutions than "just" DetNet (not to diminish DetNet any way ;-).
> >
> >The one example use case that is IMHO like DetNet missing a solid 
> >definition how to do this is exactly is the often as "live-life" 
> >described dual forwarding of IP multicast traffic in network devices:
> >
> >IMHO, IP multicast can actually help to bring down the price in 
> >products to support good live-live forwarding functions because in IP 
> >multicast this method would have applicability to more traffic than in unicast:
> >
> >In Unicast (e.g.: DetNet), the network cost is simply factor 2 in 
> >bandwidth (high level, there are arguments to bring down that perceived cost).
> >In IP multicast it actually a cost-saving: Consider you loose a 
> >packet that would be replicated to 10,000 users. And you need to 
> >protect againsts that loss. All other approaches such as other forms 
> >of FEC or ARQ (retransmission) have a relatively higher cost of bandwidth and operations than live-live.
> >
> >Even though there are various vendor options to support live-live for 
> >multicast, to the best of my knowledge, there are no implementations 
> >where the replication/elimination (often called "merge" or 
> >"de-duplication") is performed by sequence numbers in network 
> >devices. But i have not tracked this for a few years across vendors. 
> >It has been demonstrated to be IMHO easily possible in HW forwarding 
> >planes with sufficiently intelligent HW forwarding planes as early as 
> >maybe 8..10 years back and should be easily be implementable in e.g.: 
> >P4 as an example. The only instance of IETF work related to this is 
> >IMHO RFC7431, but it does not include the actual sequence number 
> >based duplication/de-duplication but only resorts to deplication 
> >based on state based switchover. Something which is a common technique in netork such a pseudowire redundancy and others if i remember correctly.
> >
> >mechanism how our retirement savings are gambled with by the 
> >financial industry - since at least hmm.. 1997 i think: market data 
> >networks. No single packet must be lost in those networks and IP 
> >multicast solutions with live-live provided that now for 20 years.
> >
> >to support the dual-path transmission. But to the best of my 
> >knowledge, they leave the duplication/de-duplication all to the application side.
> >The same is true for linear TV video, where SMPTE receiver functions 
> >do support this sequence number de-duplication as a standard function.
> >functions is IMHO best defined based on the visible parameter of its 
> >behavior at the operator side (e.g.: Yang model) because for any such 
> >high-value service you need to have some operational parameter such 
> >as measured partially-lost/fully-lost/late etc. packets, and a 
> >definition how to measure those pretty much requires a modelling of 
> >the underlying duplicate elimination model. Maybe we could even 
> >To make it widely applicable, the specification should really only 
> >refer to application of this forwarding plane function to 
> >packet-flows identified by other means (aka: any other form of 
> >looking up/identifying packet flows) and sequence numbers at a 
> >defineable offset into packets. Therefore staying agnostic to any 
> >specific choosen encap to carry the sequence numer. L2TPv3, RTP, any 
> >form of other L2/MPS/etc. encap i don't know - does not matter.
> >
> >one reference encap option that would allow to deploy a fully 
> >standardized version of this (MPLS is fine given where SPs stand today ;-)).
> >

--
---
[email-masked]

---

From: Stewart Bryant <[email-masked]>
Date: Mon, 13 Nov 2017 02:40:55 +0000

Yes, that is a good point. We need to distinguish between an 
encapsulating replicator and an in-network replicator.

An in-network replicator is stateless.

An encapsulating replicator is stateful.

- Stewart

---

From: Balázs Varga A <[email-masked]>
Date: Mon, 13 Nov 2017 02:45:12 +0000

Hi,
> An encapsulating replicator is stateful.
Not always. An ingress PE re-using the source attached S/N can be stateless.
It is stateful only if it generates the S/N.
Cheers
Bala'zs

-----Original Message-----
From: detnet [mailto:[email-masked]] On Behalf Of Stewart Bryant
Sent: 2017. november 13. 10:41
To: Toerless Eckert <[email-masked]>
Cc: Norman Finn <[email-masked]>; [email-masked]
Subject: Re: [Detnet] Some thoughts on Replication and Elimination

Yes, that is a good point. We need to distinguish between an encapsulating replicator and an in-network replicator.

- Stewart

On 13/11/2017 02:18, Toerless Eckert wrote:
> Its not sure to me whether thats a valid assumption or whether 
> creation of secquence number for specifically identified classes of 
> packets and attaching the sequence number to the packet is not also a 
> feature often considered novel especially in accelerated HW forwarding plane.
>
> Cheers
>      Toerless
>
>> replicator (whether replicating multicast or unicast traffic) uses 
>> the same packet forwarder technology as multicast. This technology is 
>>> products to support good live-live forwarding functions because in 
>>> IP multicast this method would have applicability to more traffic than in unicast:
>>>
>>> Even though there are various vendor options to support live-live 
>>> for multicast, to the best of my knowledge, there are no 
>>> implementations where the replication/elimination (often called 
>>> "merge" or "de-duplication") is performed by sequence numbers in 
>>> network devices. But i have not tracked this for a few years across 
>>> vendors. It has been demonstrated to be IMHO easily possible in HW 
>>> forwarding planes with sufficiently intelligent HW forwarding planes 
>>> as early as maybe 8..10 years back and should be easily be 
>>> implementable in e.g.: P4 as an example. The only instance of IETF 
>>> work related to this is IMHO RFC7431, but it does not include the 
>>> actual sequence number based duplication/de-duplication but only 
>>> resorts to deplication based on state based switchover. Something which is a common technique in netork such a pseudowire redundancy and others if i remember correctly.
>>>
>>> Doing sequence number based live-live has of course been the 
>>> standard mechanism how our retirement savings are gambled with by 
>>> the financial industry - since at least hmm.. 1997 i think: market 
>>> data networks. No single packet must be lost in those networks and 
>>> IP multicast solutions with live-live provided that now for 20 years.
>>>
>>> behavior at the operator side (e.g.: Yang model) because for any 
>>> such high-value service you need to have some operational parameter 
>>> such as measured partially-lost/fully-lost/late etc. packets, and a 
>>> Of course we would need to have in the same or another draft at 
>>> least one reference encap option that would allow to deploy a fully

---

From: Toerless Eckert <[email-masked]>
Date: Mon, 13 Nov 2017 03:47:50 +0100

> Sequence numbers are generated: 

Right. This is the case where forwarding plane flexibility will become handy because apps
are unlikely to generate sequence number in a "network friendly" way like MPLS header, but more inside
some RTP header for example. So even if the netork only needs to do the dedup, if we want to
support the source sequecne number generation, we have to think beyond our network centric view
for encaps.

> - at the ingress UNI (if S/N not generated by the source or it cannot be (re-)used by the ingress PE) 

Right. Thats where we would do the "network friendly sequence number headers".

Either way, let me know what/if you think is already done in network devices wrt. insertion
of sequence number so we as a WG get a better overview how novel it still is.

Cheers
    Toerless

-- 
---
[email-masked]

---

From: Toerless Eckert <[email-masked]>
Date: Mon, 13 Nov 2017 03:49:49 +0100

> It is stateful only if it generates the S/N.

So for example copy RTP sequence number into new network encap with same sequence number ?
Duplication of sequence number header might be beneficial to reduce the requirements against
the dedup side or even intermediate monitoring nodes that would not support arbitrary
app-generated sequence number headers.

Cheers
    Toerless

-- 
---
[email-masked]

---

From: Balázs Varga A <[email-masked]>
Date: Mon, 13 Nov 2017 03:24:59 +0000

Hi Stewart,

Some clarification questions on draft for better understanding:
>   Similarly one copy of a packet may be subjected to congestion discard
What do You mean here by congestion discard? Detnet flows are assuming "zero congestion loss".
See e.g., draft-ietf-detnet-architecture-04, chapter 3.2.1. Congestion protection OR
draft-finn-bounded-latency-00, chapter 4.2. Achieving zero congestion loss

>   Elimination may be configured to restrict the number of copies of a
>   packet to one, or it may permit a larger number of copies to proceed.
Permit a larger number of copies to proceed is further increase the number of states for the eliminator. 
Why not assume an "elimintaion+replication" combination function if multiple copies required after 
such a (partial-)elimination?

>    Elimination in a network where there are a non-trivial number of
>   detnet flows will require access to data-structures that cannot be
What do You mean by "non-trivial number of detnet flows"? There is always a path setup/resource reservation in 
the network before starting a detnet flow. So the number of detent flows must be known.
  
Thanks & Cheers
Bala'zs

-----Original Message-----
From: detnet [mailto:[email-masked]] On Behalf Of Stewart Bryant
Sent: 2017. november 12. 7:10
To: [email-masked]
Subject: [Detnet] Some thoughts on Replication and Elimination

I have written them up in draft format, although whether I upload the draft or not is to be determined. However I though that there was more to the issue than a simple email would easily handle.

However in in view of the uploading embago, and considering the shortness of the draft, I am posting it to the list so that it can be discussed.

- Stewart

---

From: Norman Finn <[email-masked]>
Date: Tue, 14 Nov 2017 06:37:12 +0000

Packet replication is a separate feature from bounded latency / 0 congestion loss.  You can use replication without bounded latency, and vice versa.  Of course, if your network is more than lightly loaded, packet replication as a reliability tool is likely to backfire.  But, they are separate.

-- Norm

________________________________________
From: detnet [[email-masked]] on behalf of Balázs Varga A [[email-masked]]
Sent: Sunday, November 12, 2017 7:24 PM
To: Stewart Bryant; [email-masked]
Subject: Re: [Detnet] Some thoughts on Replication and Elimination

Hi Stewart,

Thanks & Cheers
Bala'zs

-----Original Message-----
From: detnet [mailto:[email-masked]] On Behalf Of Stewart Bryant
Sent: 2017. november 12. 7:10
To: [email-masked]
Subject: [Detnet] Some thoughts on Replication and Elimination

- Stewart