Subject: Re: [6tsch] [Roll] comments on draft-phinney-roll-rpl-industrial-applicability

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Fri, 1 Mar 2013 14:58:16 +0000

Dear Michael:

[Pascal] Thanks for the comments : )

{this email was written in several sittings over 10 days. Probably I missed some stuff}

Section 1.3 says:
 1.3.  Out of scope requirements
   This applicability statement does not address requirements related to
   wireless LLNs employed in factory automation and related
   applications.

So, as I understand things, this document applies to *plants*, but not
automated factories.   Forgive me for my ignorance, but I guess this
means it applies to a oil refinery or chemical plant, but not to a car manufacturer. (Even though, we call them "auto plants")

I'm just trying to understand where the line is and why.

[Pascal] You are correct. The industry makes a difference between Factory Automation (discrete manufacturing of individual products) and Process Controls (continuous production). These are really 2 different activities, and Factory Automation may require much stricter behaviors for very rapid operations.
At ISA, WG 11 has standardized ISA100.11a for Process Control, and the information you find in the draft leverages that experience. WG16 has been working on Factory Automation.

====

>The domain of applicability for the RPL protocol may include all
>   phases but the Normal Operation phase, where the bandwidth allocation
>   and the routes are usually optimized by an external Path Computing
>   Engine (PCE), e.g. an ISA100.11a System Manager.

So, RPL could be used during phases P1, P2, P4, P5 and P6, but not during P3, (Normal Operation), unless a new OF was defined that interacted with the PCE?

[Pascal] Roughly yes, and this is the work we are starting at 6TSCH. In more details, there will be flows that can use distributed routing but critical flows will need the PCE.

(Last time I read the document, I was sure it said that LLNs could be used only during P3)

Given that, I guess the "X" in figure 1 means that RPL *can* be used?

section 2.1.2 then says:
   For factory automation networks, the basic communications cycle for
   control is typically much faster, on the order of 100 Hz or more.

but, I thought factory automation was out of scope?

[Pascal] Yes, still this is good info. 

=====

section 2.1.3:
I think that I'm confused about Source-Sink(SS) vs Peer-to-multipeer(P2MP) given that 2.1.3 says that the sink is usually a multicast address.
Is the distinction that in P2MP multiple packets are unicast?
Or is the distinction that P2MP is primary from outside the LLN, downward into the LLN, while SS is within the LLN?

[Pascal] let me quote Tom here:
"

There are three distinct paradigms used for the bulk of  industrial automation and control communication:

1) publish/subscribe, where an association exists between one source and one or a few recipients, all of which are identifiable, where publication is periodic within strict time limits (though sometimes instances are suppressed due to source compression of the stream). When pub/sub is used for closed-loop control, it is subject to timing attacks where deliberately induced message delivery jitter can destabilize a control loop. Sequentiality and non-loss are insufficient to detect this; it must be based on source time stamping , such as transport time-of-origination time stamping, that is checked at the receiver, or the equivalent, to discard anything that is not received within the proper cycle. In essence there is a fixed-acceptable-delay pipeline between the publisher and the subscriber, generally with a transit delay less than twice the publication period. COAP does not appear to be suitable for this communication paradigm without additional augmentation.

2) source/sink, where any of a number of sources logically multicasts (however that is implemented) to a predefined group address for the sinks, where the membership in that group is time-varying and unknown to the sources. Transport time-of-origination time stamps here are used for message authentication and perhaps in conjunction with source compression of related-interval time stamps in the application messaging. Arrival order and timing is of relatively little interest here, as is message loss detection (except for whatever impact that latter might have on security alerting). 

3) client/server or peer/peer, where all of the traditional properties hold and the specialized ones listed above generally do not.

"
My understanding is that this is more an upper layer paradigm then a routing issue. Pub/sub relates to periodic data that can be buffered to be read asynchronously, like the periodic publication of a measurement. Tom?

======

Or is it that the source is always on the LLN, and the sink is always elsewhere, and the fact that it's a multicast/anycast destination is just a question of how to implement redundancy in the data collection systems? 

[Pascal] Not just redundancy. You might have a copy for a local actuator and a copy for a supervisory system, a log...

page 13, section 2.1.4: says: 

   With rare
   exception, the control algorithms used with PS messaging in the
   process automation industries - those managing continuous material
   flows - rely on fixed-period sampling, computation and transfer of
   outputs, while those in the factory automation industries - those
   managing discrete manufacturing operations - rely on bounded delay
   between sampling of inputs, control computation and transfer of
   outputs to physical actuators that affect the controlled process.

it seems that there are possibly two completely different industrial requirements here which can not, and should not, be satisfied with a single DAG.  Are two documents appropriate here?

[Pascal] It is more than a different DAG, it is a different plant... I read your question as do we want an additional draft for factory automation? 

=====

>   Note: Although there are known patent applications for duocast and
>   N-cast, at the time of this writing the patent assignee, Honeywell
>   International, has offered to permit cost-free RAND use in those
>   industrial wireless standards that have chosen to employee the
>   technology, under a reciprocal licensing requirement relative to that
>   use.  

Can we get an official IPR statement via:
    http://www.ietf.org/ipr/file-disclosure

on this?  I don't think it is particularly relevant to the document, but perhaps I could be wrong.

you need to update [I-D.ietf-roll-rpl] to RFC6550.

[Pascal] Done in 03 : )

4.3.1 says:

> Because the actual link capacity depends on the particular link 
> technology used within an industrial automation network deployment, 
> the Trickle parameters are specified in terms of the link's maximum 
> capacity for conveying link-local multicast messages.  If the link can 
> convey m link-local multicast packets per second on average, the 
> expected time it takes to transmit a link-local multicast packet is 
> 1/m seconds.

I am pleased that a formula is being provided, but I am uncomfortable that this document is not more specific to actual link technology used.
Can this document pick 1-2 actual link technologies and deal with them?

[Pascal] We are working on those concepts at 6TSCH, ccing the ML.

========

5.:
  > The possibility of dynamically updating the metrics in use in the
  > network as well as the frequency of network updates allows deployment
  > characteristics (e.g., network density) to be discovered during
  > network bring-up and to be used to tailor network parameters once the
  > network is operational rather than having to rely on precise pre-
  > configuration.  This also allows the network parameters and the
  > overall routing protocol behavior to evolve during the lifetime of
  > the network.

I don't know how this discovery is going to occur.
Someone trying to build a sensor that is going to satisfy an RFP for devices to work in a network envisioned by this document needs to know
what to include in their code base.   As the code space is very finite,
they need to know exactly what is important here.

[Pascal] Finding the values for parameters is more of a deployment issue, so I read your question as 'which parameters are critical for the device to operate in industrial environment and which range for those parameters'. Again, work at 6TSCH may help.

Cheers,

Pascal

---

From: Michael Richardson <[email-masked]>
Date: Fri, 01 Mar 2013 13:34:23 -0500

>>>>> "pthubert" == pthubert  <Pascal> writes:
    pthubert> [Pascal] Thanks for the comments : )

    pthubert> [Pascal] You are correct. The industry makes a difference
    pthubert> between Factory Automation (discrete manufacturing of
    pthubert> individual products) and Process Controls (continuous
    pthubert> production). These are really 2 different activities, and
    pthubert> Factory Automation may require much stricter behaviors for
    pthubert> very rapid operations.  At ISA, WG 11 has standardized

I guess this means that individual products are things which can come
out of a production step, and then accumulate before going into the next
step. (The kind of thing Goldratt warns us against having anywhere other
than at a CCR)

    pthubert> So, RPL could be used during phases P1, P2, P4, P5 and P6,
    pthubert> but not during P3, (Normal Operation), unless a new OF was
    pthubert> defined that interacted with the PCE?

    pthubert> [Pascal] Roughly yes, and this is the work we are starting
    pthubert> at 6TSCH. In more details, there will be flows that can
    pthubert> use distributed routing but critical flows will need the
    pthubert> PCE.

I think that we need to think about:
1) are the P1/P2/P4/P5 and P6 phases distinct in some way?
   I think that there are significant security differences between them,
   and the number of type of DAGs that need to be constructed may very
   a lot. This has implications that intermediate (storing!) nodes may
   have to have much large capacities for DAGs during non-P3 than they
   do for P3.  Or not...
   My examination of RPL code to date suggests that some of what lets it
   fit into the smallest devices is because they support only a single DAG.
   (no tables, no searches, no code to sort A from B...)

2) how do you bootstrap *securely* into P3 operation?
   Does this require a rekey?  It seems to me that it does.

    pthubert> 1) publish/subscribe, where an association exists between
    pthubert> one source and one or a few recipients, all of which are
    pthubert> identifiable, where publication is periodic within strict

So, layer-6 multicasts carried on layer-3 unicast to known
subscribers... 

    pthubert> 2) source/sink, where any of a number of sources logically
    pthubert> multicasts (however that is implemented) to a predefined

1:N multicast

    pthubert> 3) client/server or peer/peer, where all of the
    pthubert> traditional properties hold and the specialized ones
    pthubert> listed above generally do not.

and this is 1:1

    pthubert> page 13, section 2.1.4: says:

    pthubert>    With rare exception, the control algorithms used with
    pthubert> PS messaging in the process automation industries - those
    pthubert> managing continuous material flows - rely on fixed-period
    pthubert> sampling, computation and transfer of outputs, while those
    pthubert> in the factory automation industries - those managing
    pthubert> discrete manufacturing operations - rely on bounded delay
    pthubert> between sampling of inputs, control computation and
    pthubert> transfer of outputs to physical actuators that affect the
    pthubert> controlled process.

    pthubert> it seems that there are possibly two completely different
    pthubert> industrial requirements here which can not, and should
    pthubert> not, be satisfied with a single DAG.  Are two documents
    pthubert> appropriate here?

    pthubert> [Pascal] It is more than a different DAG, it is a
    pthubert> different plant... I read your question as do we want an
    pthubert> additional draft for factory automation?

So, factory automation and plant control are not present in the same
building.  When someone comes along with factory automation expertise,
they will need to write a draft.

I'm not sure why factory automation is mentioned again.
Let's just say:

Tthe control algorithms used with PS messaging in the process automation
industries - those managing continuous material flows - rely on fixed-period
sampling, computation and transfer of outputs.   As a result.. X, Y, Z.

    >> Note: Although there are known patent applications for duocast
    >> and N-cast, at the time of this writing the patent assignee,
    >> Honeywell International, has offered to permit cost-free RAND use
    >> in those industrial wireless standards that have chosen to
    >> employee the technology, under a reciprocal licensing requirement
    >> relative to that use.

    pthubert> Can we get an official IPR statement via:
    pthubert> http://www.ietf.org/ipr/file-disclosure

    pthubert> on this?  I don't think it is particularly relevant to the
    pthubert> document, but perhaps I could be wrong.

Not sure why it is mentioned then.

    pthubert> you need to update [I-D.ietf-roll-rpl] to RFC6550.

    pthubert> [Pascal] Done in 03 : )

    pthubert> 4.3.1 says:

    >> technology used within an industrial automation network
    >> deployment, the Trickle parameters are specified in terms of the
    >> link's maximum capacity for conveying link-local multicast
    >> messages.  If the link can convey m link-local multicast packets
    >> per second on average, the expected time it takes to transmit a
    >> link-local multicast packet is 1/m seconds.

    pthubert> I am pleased that a formula is being provided, but I am
    pthubert> uncomfortable that this document is not more specific to
    pthubert> actual link technology used.  Can this document pick 1-2
    pthubert> actual link technologies and deal with them?

    pthubert> [Pascal] We are working on those concepts at 6TSCH, ccing
    pthubert> the ML.

But, wait. 6tsch is important for P3 only.
The other phases need to have something written down.
I think it's pretty important to have some concrete numbers for some
actual technologies that actually are being deployed.    Without that,
it's pretty hard for academics to run simulations, or for people
thinking about RPL extensions in field X to think how this might work
(or not) in another field they aren't an expert in.

Also, we need that set of layer 2s to be listed so that the security
requirements of layer 2 can be expressed.  Again, non-P3 vs P3 might
require a rekey.

    pthubert> 5.:
    >> network as well as the frequency of network updates allows
    >> deployment characteristics (e.g., network density) to be
    >> discovered during network bring-up and to be used to tailor
    >> network parameters once the network is operational rather than
    >> having to rely on precise pre- configuration.  This also allows
    >> the network parameters and the overall routing protocol behavior
    >> to evolve during the lifetime of the network.

   mcr> I don't know how this discovery is going to occur.
   mcr> Someone trying to build a sensor that is going to satisfy
   mcr> an RFP for devices to work in a network envisioned by this
   mcr> document needs to know what to include in their code base.
   mcr> As the code space is very finite, they need to know
   mcr> exactly what is important here.

    pthubert> [Pascal] Finding the values for parameters is more of a
    pthubert> deployment issue, so I read your question as 'which
    pthubert> parameters are critical for the device to operate in
    pthubert> industrial environment and which range for those
    pthubert> parameters'. Again, work at 6TSCH may help.

Yes, that's what I'm saying.

The text talks about "dynamically updating the metrics in use"

That says to me that it's not about the values of the metrics, but in
changing which metrics are used.

-- 
]               Never tell me the odds!                 | ipv6 mesh networks [ 
]   Michael Richardson, Sandelman Software Works        | network architect  [ 
]     [email-masked]  http://www.sandelman.ca/        |   ruby on rails    [

---

From: Tom Phinney <[email-masked]>
Date: Fri, 01 Mar 2013 12:42:20 -0700

Apologies to those who receive this e-mail on
more than one list.

First, the traditional industry differentiation between "factory
automation" and "process control" is that factory automation is
focused on discrete manufacturing steps (and thus is often
referred to as "discrete parts manufacturing") while process
control involves continuous processing of fluids and quasi-fluids
(e.g., sand, cereal and other materials whose flow rate can be
measured by a mass flowmeter).

Many real-world plants use hybrids of these two manufacturing
styles. For example, a soup canning plant, or drink bottling plant
(e.g., a brewery), or a dairy that bottles milk and packages ice
cream, or a pharmaceutical plant, or a car tire manufacturing
plant, all use continuous-flow manufacturing processes (process
control) to make the raw product in batches, and discrete part
manufacturing processes (factory automation) to package the
product. This hybrid mode of operation is often referred to as
"batch processing" and has its own terminology and set of
standards that build on those of the other two manufacturing
regimes. ISA, the Industrial Society for Automation, has developed
standards that address both continuous process control and batch
processes that are fed materials from a continuous process. (Most
ISA standards then become IEC standards.)

Second, the distinction between peer/peer, client/server, publish/subscribe
and source/sink is as follows:

1a) peer/peer is a 1:1 relationship of
known correspondents, usually connected
(i.e., a longer-term association with coordinated
residual state information at both ends).
This relationship can be symmetric (i.e., federation) or
asymmetric (i.e., client/server).

1b) client/server is an asymmetric peer-peer relationship of "known" correspondents
in which the client is always the initiator of the relationship.
In cases where the provided service is transactional, the
association often is more transient than in a symmetric peer-peer
relationship. In this case the server may be a member of a server
farm, in which case the correspondent is known by role, not device
ID.

2) publish/subscribe is a 1:N relationship of known
correspondents, which is best implemented as a multi-peer
connection (i.e., a longer-term
association with coordinated residual state information at the
endpoints). Some industrial communication technologies,
such as IEC 61158 Type 1 (Foundation Fieldbus H1) provide multi-peer
connections within the lower four comm layers to directly support
such operation. Timeliness is usually important in the
publish/subscribe relationship, whether it be the timeliness of
daily newspaper delivery or the timeliness of information delivery
between devices acting in concert in closed-loop continuous control.
As a consequence, untimely retransmission is not of significant
value.

3) source/sink is a M:N relationship of unknown correspondents,
which is best implemented by connectionless lower layer services. In
the automation environment it is used primarily for reporting of
alerts, which include a) device-local events and b) alarms about the
monitored process. Alerts are reportable by every device that is
involved in sensing and controlling the process -- the M of M:N --
and are reported to all devices that are interested in receiving
such reports -- the N of M:N. That latter set includes all human
operator workstations that are focused on the monitored process, all
historians that are tracking information in the monitored process,
etc. There is never a need for comm-layer-initiated retransmission
of such information; other application-layer mechanisms come into
play to retrieve recent reporting history from individual members of
the M set when loss of a report is detected an an N endpoint. To
make such loss detection and bulk recovery possible, each alert
source includes a local monotonically-increasing sequence number in
each alert report.

So, in summary,

1a) peer/peer is 1:1 and typically connected;

1b) client/server is 1:1 and often connected, particularly when the
association is longer than a single transaction,

2) publish/subscribe is 1:N with known N, and connected in a
long-term relationship;

3) source/sink is M:N with unknown M and N, and connectionless.

Typically 1a) and 1b) use unicast in each direction; 2) uses unicast
in the subscriber-to-publisher direction and multicast to a
publication-specific group address in the publisher-to-subscriber
direction; and 3) uses multicast to a group address identifying the
reported information scope (e.g., "unit_N_alerts").

-Tom

=====

On 2013.03.01 11:34, Michael Richardson wrote:

1:N multicast

and this is 1:1

---

From: Michael Richardson <[email-masked]>
Date: Fri, 01 Mar 2013 15:03:25 -0500

>>>>> "Tom" == Tom Phinney <[email-masked]> writes:
    Tom>    Apologies to those who receive this e-mail on more than one
    Tom> list.  First, the traditional industry differentiation between
    Tom> "factory automation" and "process control" is that factory
    Tom> automation is focused on discrete manufacturing steps (and thus
    Tom> is often referred to as "discrete parts manufacturing") while
    Tom> process control involves continuous processing of fluids and
    Tom> quasi-fluids (e.g., sand, cereal and other materials whose flow
    Tom> rate can be measured by a mass flowmeter).  Many real-world
    Tom> plants use hybrids of these two manufacturing styles.  

So, given that many plants are hybrids, are you saying that this
document applies only to the process control parts of the plant?

Or that hybrid plants are also out of scope?

If there are scope constraints, what exactly are they? 

What happens in a hydrid plant might be significantly more complicated,
as it involves two kinds of networks; if there is a desire to have the
different pieces interconnected.

---

From: Tom Phinney <[email-masked]>
Date: Fri, 01 Mar 2013 14:32:00 -0700

The continuous and discrete manufacturing parts
of hybrid manufacturing plants are ALWAYS connected; it would be
pointless to build such a plant if that were not the case.

It is useful to consider the differences. Continuous processes use
closed-loop control based on fixed-period process sampling and
outputs to process actuators. The highest frequency of such
control loops is usually 4 H, though a few compressor surge
control loops that operate at a 9 Hz to 11 Hz rate may also be
present. These process control systems are potential users of
wireless communications because their fixed timing is compatible
(in theory) with TSCH-like time schedules.

It is important to understand that the publish-subscribe messaging
used for process automation does not require 100% delivery. The
control algorithms traditionally are designed to tolerate the loss
of three or fewer consecutive messages before receiving another
message in the series. However, typically loss of four consecutive
messages triggers the equipment to shift to a fallback control
strategy. It is this statistic that leads to use of duocast (with
its approximate squaring of message loss rates) for high-speed
loops, since most plant managers will not tolerate more than one
such shift to a fallback control strategy per year, per plant, and
there can be thousands of 1 Hz and 4 Hz control loops operating in
the plant.

Discrete processes use a different form of control that is cyclic
but not periodic. This is usually implemented in PLCs
(programmable logic controllers) that execute a
variable-execution-duration program that repeats as soon as it
finishes each iteration, leading to cyclic but aperiodic behavior.
The duration of each execution cycle is usually a few ms,
typically 3 ms to 10 ms, with a perhaps 50% maximum variance in
cycle-to-cycle.execution time. The
messaging involved in such automation typically require higher
data rates than those offered by IEEE 802.15.4, and any wireless
messaging itself would typically be single-hop, from a
router/local-controller that is part of the manufacturing device
to multiple wireless backbone routers arranged high on opposite
walls in a roofed factory. High-data-rate optical signaling would
be a good match for this communication.

The variance in cycle-to-cycle times provides a significant
challenge for comm technologies based on scheduled pairing of
senders and receivers. That, coupled with the typically single-hop
aspect of the communication, makes it likely that a non-TSCH
approach will be optimal for such systems.

The communication between the continuous process controllers of a
plant and the discrete automation controllers typically occurs on
a 100 Mbit/s or faster backbone comm link. Wireless is not
involved, and there is no reason to anticipate that it will be in
the future. Since the physical systems that implement the
continuous and discrete processes are interconnected (e.g., pipes
with valves connected to bottle handling-and-filling apparatus in
a soda bottling plant), wired interconnection between backbone
controllers is typically not a significant issue.

I hope that the above discussion helps understand the differences
in operating mode, timescales, comm needs and other high-level
aspects of the continuous and discrete portions of a hybrid plant.
There are others receiving this e-mail who are much more familiar
with such plants and processes than I, so if I have erred
substantially, I trust that they will post a correction to this
list.

-Tom

=====

On 2013.03.01 13:03, Michael Richardson wrote:

_______________________________________________
6tsch mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/6tsch