Subject: [6tsch] priority and priority

From: Thomas Watteyne <[email-masked]>
Date: Thu, 7 Mar 2013 23:34:44 -0800

All,

I believe that we have been discussing two different notions of priority:
- *Packet Priority*: Assuming an existing data flow in a network, some
packets are marked as "high priority", the rest as "low priority". All
packets are part of the same flow, but high priority packets are treated
with more urgency that low priority packets. This translates into a set
of queuing rules. This is the notion illustrated by Pascal in his "traffic
lights in a city" analogy (where high priority packets are police cars
passing other vehicles) and used in Qin's mux/i-mux.
- *Flow Priority*, which is slightly different. In a network, multiple
classes of traffic co-exist. This results in multiple data flows. Different
data flows have a different combination of destination, priority, bandwidth
and latency. A network-wide policy defines whether data flows are entirely
independent, or if some over-flowing is allowed. In TSCH, a flow translated
into a slotframe. In case of contention (e.g. at a given slot a mote can
send a packet from each flow), flow priority arbitrates which packet goes
first. Note that this is entirely within the TSCH spec.

These two definitions are complementary: different flows can exist in a
network, but inside a flow, some high priority packets can "pass" other
packets. The network policy defines how those two interact; e.g. a
highest-priority packet can be allowed to "jump" into another flow, if
needed.

We need to define how this materializes in a network. Here is a proposal:

Each data flow turns into a TSCH slotframe. The scheduling entity populates
the slots in the slotframe in such as way that the slots "point to" some
destination. In a typical case, slots in a slotframe are equivalent: to end
up at mote A, take any slot in slotframe 1. This allows us to use
label-switching approach, which I'm sure will pop up in some future draft.
We can end up with different slotframes for different destinations, or for
different types of traffic to the same destination.

Optionally, packets can contain some priority field (probably DSCP, as per
http://tools.ietf.org/html/draft-svshah-tsvwg-lln-diffserv-recommendations-00?)
which serves as the "high priority" marker. A policy then regulates how
this translates to queuing and flow equivalence rules.

DSCP can also be used by the higher layer to indicate to which flow this
packet pertains, and with what (optional) priority. Upon receiving a packet
from a higher layer, the 6tsch layer could inspect the DSCP and some set of
rules would tell him which flow to send it on. Once it is in a flow, the
full DSCP is not needed, and there is lots of opportunity to elide some
information while the packet is on the LLN. I'm hoping that Shitanshu a.o.
can help decide on the applicability of this.

Thomas

---

From: "Alfredo Grieco" <[email-masked]>
Date: Fri, 8 Mar 2013 09:56:11 +0100

Dear Thomas and all,
 
I think that the adoption of label switching approaches is totally
appropriate in this context and can help solving several issues, including
priority.
 
As a matter of fact, the label (as in MPLS) can encode many different
properties: priority, destination node, kind of application, and whatever we
want.
 
The point is we should adopt some lightweight label dissemination mechanism
for the LLN context, but I am sure that approaches based on RSVP (one among
others) could surely fit well.
 
Cheers
 
Alfredo
 
Da: [email-masked] [mailto:[email-masked]] Per conto di
Thomas Watteyne
Inviato: Friday, March 08, 2013 8:35 AM
A: IETF 6TSCH
Oggetto: [6tsch] priority and priority
 
All,
 
I believe that we have been discussing two different notions of priority:
- Packet Priority: Assuming an existing data flow in a network, some packets
are marked as "high priority", the rest as "low priority". All packets are
part of the same flow, but high priority packets are treated with more
urgency that low priority packets. This translates into a set of queuing
rules. This is the notion illustrated by Pascal in his "traffic lights in a
city" analogy (where high priority packets are police cars passing other
vehicles) and used in Qin's mux/i-mux.
- Flow Priority, which is slightly different. In a network, multiple classes
of traffic co-exist. This results in multiple data flows. Different data
flows have a different combination of destination, priority, bandwidth and
latency. A network-wide policy defines whether data flows are entirely
independent, or if some over-flowing is allowed. In TSCH, a flow translated
into a slotframe. In case of contention (e.g. at a given slot a mote can
send a packet from each flow), flow priority arbitrates which packet goes
first. Note that this is entirely within the TSCH spec.

 
 
Optionally, packets can contain some priority field (probably DSCP, as per
http://tools.ietf.org/html/draft-svshah-tsvwg-lln-diffserv-recommendations-0
0?) which serves as the "high priority" marker. A policy then regulates how
this translates to queuing and flow equivalence rules.

Thomas

---

From: Michael Richardson <[email-masked]>
Date: Fri, 08 Mar 2013 21:55:28 -0500

>>>>> "Thomas" == Thomas Watteyne <[email-masked]> writes:
    Thomas> I believe that we have been discussing two different notions
    Thomas> of priority: - *Packet Priority*: Assuming an existing data
    Thomas> flow in a network, some packets are marked as "high
    Thomas> priority", the rest as "low priority". All packets are part
    Thomas> of the same flow, but high priority packets are treated with
    Thomas> more urgency that low priority packets. This translates into

the word 'flow' is ambiguous.
    It IETF land, a flow is a 2 or 3-tuple of
    (source,dest,sometimes-protocol-sometimes-flow-label) 
    (a microflow is a 5-tuple, SA,DA,protocol,source-port,dest-port)

How can a flow have high priority packets and low priority packets?

Why are these two types of packets using the same DODAG?

High priority packets presumeably require lower latency, and therefore
it might make sense to use a more direct route through a more costly
node.  Or, if a particular PAN is congested, they might get priority
access to the slots, while lower priority traffic might have to take a
more circuitous route around the edges...

    Thomas> DSCP can also be used by the higher layer to indicate to
    Thomas> which flow this packet pertains, and with what (optional)
    Thomas> priority. Upon receiving a packet from a higher layer, the
    Thomas> 6tsch layer could inspect the DSCP and some set of rules
    Thomas> would tell him which flow to send it on. Once it is in a

Well, just think instead that 6tsch could "inspect" the DSCP, say that
the 6tsch API will provide an way for the layer 3 to express the
DSCP-equivalent. 

-- 
Michael Richardson
-on the road-

---

From: Thomas Watteyne <[email-masked]>
Date: Fri, 8 Mar 2013 22:33:22 -0800

Michael,

I agree that the term flow is almost as overloaded as link or path. This is
not a final term and we will have to come up with a better term.
Suggestions?

If I get your point correctly, you are suggesting that there should be no
packet priorities, only flows priority. In a TSCH context, this would
translate into a slotframe per flow, where flow also incorporates the
notion of priority. Correct? This is a very important discussion we started
over the phone earlier today, but which we will continue Wednesday in
Orlando. I hope you can be there.

Thomas

On Fri, Mar 8, 2013 at 6:55 PM, Michael Richardson <[email-masked]>wrote:

> _______________________________________________
> 6tsch mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/6tsch
>
>

---

From: Tom Phinney <[email-masked]>
Date: Sat, 09 Mar 2013 00:36:00 -0700

I don't know what real-world use this technology is being designed
to support, but the discussion fragments quoted at the end below
seem to me to be moving away from reality.

In the industrial automation world where I spent most of my career,
it makes sense to have different inbound flows for differing
purposes. Most sensor devices would have the following flows 1), 2)
and 3):

1) a dedicated inbound flow per field device (mote) for periodic
deterministic control traffic, where the number of expected hops has
to be small when the loop rate is high;

2) a shared inbound flow for usually-infrequent alerts (i.e., device
events and process alarms), probably with at least 2-level packet
queuing priority, perhaps using CSMA/CA to provide some slight
statistical bias in channel access priority based on packet
priority;

3) a shared inbound flow for server-to-client responses, typically
in response to a program-initiated or operator-initiated action at
an engineering workstation in a control room.

Every process actuator (i.e., mote that can directly manipulate the
process) will also have another flow 4), analogous to 1) but in the
outbound direction, from the control room to the field to transfer
the computed actuator setting to the device. Such devices also have
inbound flow 1) to transfer actuator status at the same periodic
rate.

If communication problems are experienced on flow 1), it is
desirable to be able to use one or more of the other flows as a
backup mechanism for inbound control traffic, whether full
determinism can be maintained or not. Packet queuing priority
definitely makes sense here, so that such traffic can leapfrog
non-deterministic traffic in the output queue of the originating
mote and of any intermediary relaying motes. As mentioned in 2), it
may also make sense to use CSMA/CA to provide some slight
statistical bias in channel access priority based on packet
priority, at least for flows where the transaction template has a
non-null CSMA/CA sensing interval before the scheduled Data PDU
transmission of the transaction.

The reason for priority on the alerts is that process alarms from
class 1 - 2 devices, (or class 0 - 2 if class 0 uses wireless),
would usually be configured to be higher priority than those for
classes 3 - 5. (Some class 3 devices might also be assigned that
higher alarm priority.) Where safety alarms are concerned, such as
the "man down" alarm that Herman Storey mentioned on Thursday's
call, those alarms would tend to use another flow 5), similar to 2),
with a very low expected rate but with dedicated slots to provide a
clear virtual channel (to the extent possible). The time constants
for such alarms are long enough (seconds) compared to much of the
flow 1) traffic that the incremental cost of flow 5), which is so
similar to flow 2), probably would not be very much, perhaps 2% of
the total network capacity. It is tempting to just have a 3-way
CSMA/CA interval for flow 2), but the reality of the CSMA/CA
priority assessment mechanism is that it is very weak, given
physical modem delays, inter-mote timing skews and the old
hidden-node problem. Thus CSMA/CA prioritization does not seem
reliable enough to trust for safety purposes.

Of course a system will have other flows, many allocated on a demand
basis in response to some occasional need, such as firmware download
or captured waveform upload.

The basic timing of continuous control loops it that the total delay
from sensing the process to driving the actuator generally has to be
less than twice the period of the loop. In such cases the needed
control strategy is both relatively simple and relatively stable,
However, if the total input-to-output delay exceeds twice the loop
period, the control becomes much more difficult and less able to
respond to fast transients. Jitter is a problem only when a process
value message is delivered in the wrong cycle; it does not matter
whether the delivery is at the 10% or 90% point in the proper cycle.
Authenticated timestamping of the message, as occurs via the UDP
transport nonce in ISA100.11a, provides a means of discarding
process value messages that are delivered too late, thus converting
those situations to ones of message loss. That is the desired way of
handling late delivery, because the control loop is stable under
message loss but not when the values are deliberately selectively
delayed into the wrong delivery cycle. (Honeywell demonstrated years
ago that it was possible to destablize most control loops with such
deliberately jittered delivery.)

-Tom

=====

On 2013.03.08 23:33, Thomas Watteyne wrote:

Michael,

I agree that the term flow is almost as overloaded as link or
path. This is not a final term and we will have to come up with
a better term. Suggestions?

If I get your point correctly, you are suggesting that there
should be no packet priorities, only flows priority. In a TSCH
context, this would translate into a slotframe per flow, where
flow also incorporates the notion of priority. Correct? This is
a very important discussion we started over the phone earlier
today, but which we will continue Wednesday in Orlando. I hope
you can be there.

Thomas

On Fri, Mar 8, 2013 at 6:55 PM, Michael
Richardson <[email-masked]>
wrote:

How can a flow have high priority
packets and low priority packets?

---

From: Kris Pister <[email-masked]>
Date: Sat, 09 Mar 2013 09:36:58 -0800

Tom - I agree with everything that you wrote in the email below except 
the first sentence.  Perhaps I'm missing something, but it seems to me 
that your enumeration of various flows with different requirements fits 
quite well with the discussion fragment quoted at the end below.  Can 
you help me understand the differences?

My bias is that we should have
{flow priority (equating to slotframe priority) OR link priority (TX, 
RX, uni-, multi-)} AND {packet priority}.

ksjp

On 3/8/2013 11:36 PM, Tom Phinney wrote:
> I don't know what real-world use this technology is being designed to 
> support, but the discussion fragments quoted at the end below seem to 
> me to be moving away from reality.
>
> In the industrial automation world where I spent most of my career, it 
> makes sense to have different inbound flows for differing purposes. 
> Most sensor devices would have the following flows 1), 2) and 3):
> statistical bias in channel access priority based on packet priority;
> 3) a shared inbound flow for server-to-client responses, typically in 
> response to a program-initiated or operator-initiated action at an 
> engineering workstation in a control room.
>
> outbound direction, from the control room to the field to transfer the 
> computed actuator setting to the device. Such devices also have 
> inbound flow 1) to transfer actuator status at the same periodic rate.
>
> If communication problems are experienced on flow 1), it is desirable 
> to be able to use one or more of the other flows as a backup mechanism 
> for inbound control traffic, whether full determinism can be 
> maintained or not. Packet queuing priority definitely makes sense 
> here, so that such traffic can leapfrog non-deterministic traffic in 
> the output queue of the originating mote and of any intermediary 
> relaying motes. As mentioned in 2), it may also make sense to use 
> CSMA/CA to provide some slight statistical bias in channel access 
> priority based on packet priority, at least for flows where the 
> transaction template has a non-null CSMA/CA sensing interval before 
> the scheduled Data PDU transmission of the transaction.
>
> class 1 - 2 devices, (or class 0 - 2 if class 0 uses wireless), would 
> usually be configured to be higher priority than those for classes 3 - 
> 5. (Some class 3 devices might also be assigned that higher alarm 
> priority.) Where safety alarms are concerned, such as the "man down" 
> alarm that Herman Storey mentioned on Thursday's call, those alarms 
> would tend to use another flow 5), similar to 2), with a very low 
> expected rate but with dedicated slots to provide a clear virtual 
> channel (to the extent possible). The time constants for such alarms 
> are long enough (seconds) compared to much of the flow 1) traffic that 
> the incremental cost of flow 5), which is so similar to flow 2), 
> probably would not be very much, perhaps 2% of the total network 
> capacity. It is tempting to just have a 3-way CSMA/CA interval for 
> flow 2), but the reality of the CSMA/CA priority assessment mechanism 
> is that it is very weak, given physical modem delays, inter-mote 
> timing skews and the old hidden-node problem. Thus CSMA/CA 
> prioritization does not seem reliable enough to trust for safety purposes.
>
> transport nonce in ISA100.11a, provides a means of discarding process 
> value messages that are delivered too late, thus converting those 
> situations to ones of message loss. That is the desired way of 
>> I agree that the term flow is almost as overloaded as link or path. 
>> This is not a final term and we will have to come up with a better 
>> term. Suggestions?
>>
>> If I get your point correctly, you are suggesting that there should 
>> be no packet priorities, only flows priority. In a TSCH context, this 
>> would translate into a slotframe per flow, where flow also 
>> incorporates the notion of priority. Correct? This is a very 
>> important discussion we started over the phone earlier today, but 
>> which we will continue Wednesday in Orlando. I hope you can be there.
>>
>> Thomas
>>
>> On Fri, Mar 8, 2013 at 6:55 PM, Michael Richardson 
>> <[email-masked] <mailto:[email-masked]>> wrote:
>>

---

From: Tom Phinney <[email-masked]>
Date: Sat, 09 Mar 2013 12:27:13 -0700

Kris (et al),

Thank you for your reply and general concurrence.

In my post below I was objecting to the conclusion in the
originally quoted interchange, recolored in
red at the end below, that flow priority suffices. My
example flow 2) is one in which packet priority makes sense WITHIN
the flow. Note also my text in blue below,
highlighting a similar conclusion for use of flows 2) and/or 3) as
backup channels for flow 1) when flow 1) becomes too unreliable.

My post below was an attempt to point out that queuing priority
AND flow priority need not be disjoint approaches; in my opinion
they can and should both exist concurrently in systems that deal
with real-world problems, at least in the industrial and process
automation markets. They each provide some functionality that the
other lacks, giving an overall system that is superior in its
performance of critical functions to one that restricts itself to
flow priority, or to queuing priority, but without supporting
both. My post below was an attempt to sketch a common scenario in
that market where the two priority mechanisms interplay to provide
a superior system.

Perhaps my concept of flow differs from those of the original two
correspondents. If so, I would appreciate instruction/correction.
My scenario below was one where resources assigned to secondary
flows would be used when the primary flow 1) fails. If others'
notions of flow are not coupled to contracts and resource
allocation in forwarding DL and NL routers, potentially including slot prioritization among
concurrent slots, then what use is it?

-Tom

=====

On 2013.03.09 10:36, Kris Pister wrote:

Tom - I agree with everything that you wrote in the email below
except the first sentence.  Perhaps I'm missing something, but it
seems to me that your enumeration of various flows with different
requirements fits quite well with the discussion fragment quoted
at the end below.  Can you help me understand the differences?

{flow priority (equating to slotframe priority) OR link priority
(TX, RX, uni-, multi-)} AND {packet priority}.

ksjp

On 3/8/2013 11:36 PM, Tom Phinney
wrote:

I don't know what real-world use this technology is being
designed to support, but the discussion fragments quoted at the
end below seem to me to be moving away from reality.

In the industrial automation world where I spent most of my
career, it makes sense to have different inbound flows for
differing purposes. Most sensor devices would have the following
flows 1), 2) and 3):

1) a dedicated inbound flow per field device (mote) for periodic
deterministic control traffic, where the number of expected hops
has to be small when the loop rate is high;

2) a shared inbound flow for usually-infrequent alerts (i.e.,
device events and process alarms), probably with at least
2-level packet queuing priority, perhaps using CSMA/CA to
provide some slight statistical bias in channel access priority
based on packet priority;

3) a shared inbound flow for server-to-client responses,
typically in response to a program-initiated or
operator-initiated action at an engineering workstation in a
control room.

Every process actuator (i.e., mote that can directly manipulate
the process) will also have another flow 4), analogous to 1) but
in the outbound direction, from the control room to the field to
transfer the computed actuator setting to the device. Such
devices also have inbound flow 1) to transfer actuator status at
the same periodic rate.

If communication problems are experienced on flow 1), it is desirable to be able to use one or more
of the other flows as a backup mechanism for inbound control
traffic, whether full determinism can be maintained or
not. Packet queuing priority definitely
makes sense here, so that such traffic can leapfrog
non-deterministic traffic in the output queue of the
originating mote and of any intermediary relaying motes.
As mentioned in 2), it may also make sense to use CSMA/CA to
provide some slight statistical bias in channel access priority
based on packet priority, at least for flows where the
transaction template has a non-null CSMA/CA sensing interval
before the scheduled Data PDU transmission of the transaction.

The reason for priority on the alerts is that process alarms
from class 1 - 2 devices, (or class 0 - 2 if class 0 uses
wireless), would usually be configured to be higher priority
than those for classes 3 - 5. (Some class 3 devices might also
be assigned that higher alarm priority.) Where safety alarms are
concerned, such as the "man down" alarm that Herman Storey
mentioned on Thursday's call, those alarms would tend to use
another flow 5), similar to 2), with a very low expected rate
but with dedicated slots to provide a clear virtual channel (to
the extent possible). The time constants for such alarms are
long enough (seconds) compared to much of the flow 1) traffic
that the incremental cost of flow 5), which is so similar to
flow 2), probably would not be very much, perhaps 2% of the
total network capacity. It is tempting to just have a 3-way
CSMA/CA interval for flow 2), but the reality of the CSMA/CA
priority assessment mechanism is that it is very weak, given
physical modem delays, inter-mote timing skews and the old
hidden-node problem. Thus CSMA/CA prioritization does not seem
reliable enough to trust for safety purposes.

Of course a system will have other flows, many allocated on a
demand basis in response to some occasional need, such as
firmware download or captured waveform upload.

The basic timing of continuous control loops it that the total
delay from sensing the process to driving the actuator generally
has to be less than twice the period of the loop. In such cases
the needed control strategy is both relatively simple and
relatively stable, However, if the total input-to-output delay
exceeds twice the loop period, the control becomes much more
difficult and less able to respond to fast transients. Jitter is
a problem only when a process value message is delivered in the
wrong cycle; it does not matter whether the delivery is at the
10% or 90% point in the proper cycle. Authenticated timestamping
of the message, as occurs via the UDP transport nonce in
ISA100.11a, provides a means of discarding process value
messages that are delivered too late, thus converting those
situations to ones of message loss. That is the desired way of
handling late delivery, because the control loop is stable under
message loss but not when the values are deliberately
selectively delayed into the wrong delivery cycle. (Honeywell
demonstrated years ago that it was possible to destablize most
control loops with such deliberately jittered delivery.)

-Tom

=====

Michael,

I agree that the term flow is almost as overloaded as
link or path. This is not a final term and we will have to
come up with a better term. Suggestions?

If I get your point correctly, you are suggesting that
there should be no packet priorities,
only flows priority. In a TSCH context, this would
translate into a slotframe per flow, where flow also
incorporates the notion of priority. Correct? This is a very
important discussion we started over the phone earlier
today, but which we will continue Wednesday in Orlando. I
hope you can be there.

Thomas

On Fri, Mar 8, 2013 at 6:55 PM,
Michael Richardson <[email-masked]>
wrote:

How can a flow
have high priority packets and low priority packets?

---

From: Kris Pister <[email-masked]>
Date: Sat, 09 Mar 2013 12:22:34 -0800

got it.  I think that you and I are pretty closely aligned then.

ksjp

On 3/9/2013 11:27 AM, Tom Phinney wrote:
> In my post below I was objecting to the conclusion in the originally 
> quoted interchange, recolored in red at the end below, that flow 
> priority suffices. My example flow 2) is one in which packet priority 
> makes sense WITHIN the flow. Note also my text in blue below, 
> My post below was an attempt to point out that queuing priority AND 
> flow priority need not be disjoint approaches; in my opinion they can 
> and should both exist concurrently in systems that deal with 
> real-world problems, at least in the industrial and process automation 
> markets. They each provide some functionality that the other lacks, 
> giving an overall system that is superior in its performance of 
> critical functions to one that restricts itself to flow priority, or 
> to queuing priority, but without supporting both. My post below was an 
> attempt to sketch a common scenario in that market where the two 
> priority mechanisms interplay to provide a superior system.
>
> correspondents. If so, I would appreciate instruction/correction. My 
> scenario below was one where resources assigned to secondary flows 
> would be used when the primary flow 1) fails. If others' notions of 
> flow are not coupled to contracts and resource allocation in 
> forwarding DL and NL routers, potentially including slot 
> prioritization among concurrent slots, then what use is it?
>
> -Tom
> =====
>> requirements fits quite well with the discussion fragment quoted at 
>> the end below.  Can you help me understand the differences?
>>
>>>> be no packet priorities, only flows priority. In a TSCH context, 
>>>> this would translate into a slotframe per flow, where flow also

---

From: "Shitanshu Shah (svshah)" <[email-masked]>
Date: Sun, 10 Mar 2013 05:04:22 +0000

Hi Tom, All,

In general I agree with your assessment for need for packet priority.

If forwarding service required is similar to all packets for a given class, packet priority is all what is required for such services. Like traffic classes in your example for deterministic class, alerts, server-client response etc..  Treating such class of traffic at the aggregate level per packet priority (eg. Based on certain L3 or L2 level code-points) is what is needed for forwarding(queuing) service.

I also lack to understand, what is the definition of flow, is it micro-flow that has been eluded before?
For micro-flows belonging to the same service class (eg. Belonging to alert class), what kind of differentiation needed for different flows within that class? I would imagine that queuing is expensive and scarce resources in these systems as well just like it is for traditional Layer2 switches (at the least in the hardware). Thus probably very expensive to imagine as many queues as many set of flows with different priority.

However, a sort of differentiation that can be imagined at flow level may be for other parameters (but not queuing behavior). For example, limiting rate of a flow that is fed in to the queuing system.excessive traffic to be dropped or re-marked to lower priority code-point. Or during congestion drop traffic from certain flows over others.

I understand RSVP provides a way to enable integrated queuing service. At the same time, RSVP also has a way to enable reservation based on Diffserv. It is later that I am trying to highlight where packet priority is used for the forwarding decision, and flow classification may be used to constrain other parameters like rate.

Taking deterministic class in particular,
As I am not so much familiar with Industrial Automation, what I am not clear if for a given PAN, could there be different set of flows with different deterministic parameters? If there are then I can see some impact to queuing discipline and packet priority itself may not be sufficient. But I am not sure why there would be an application (PAN) with devices with different set of deterministic flows. I can also see them causing conflicts when it comes to channel reservation.

Regards,
Shitanshu

From: Tom Phinney <[email-masked]<mailto:[email-masked]>>
Reply-To: Tom Phinney <[email-masked]<mailto:[email-masked]>>
Date: Saturday, March 9, 2013 11:27 AM
To: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: Re: [6tsch] priority and priority

Kris (et al),

In my post below I was objecting to the conclusion in the originally quoted interchange, recolored in red at the end below, that flow priority suffices. My example flow 2) is one in which packet priority makes sense WITHIN the flow. Note also my text in blue below, highlighting a similar conclusion for use of flows 2) and/or 3) as backup channels for flow 1) when flow 1) becomes too unreliable.

My post below was an attempt to point out that queuing priority AND flow priority need not be disjoint approaches; in my opinion they can and should both exist concurrently in systems that deal with real-world problems, at least in the industrial and process automation markets. They each provide some functionality that the other lacks, giving an overall system that is superior in its performance of critical functions to one that restricts itself to flow priority, or to queuing priority, but without supporting both. My post below was an attempt to sketch a common scenario in that market where the two priority mechanisms interplay to provide a superior system.

Perhaps my concept of flow differs from those of the original two correspondents. If so, I would appreciate instruction/correction. My scenario below was one where resources assigned to secondary flows would be used when the primary flow 1) fails. If others' notions of flow are not coupled to contracts and resource allocation in forwarding DL and NL routers, potentially including slot prioritization among concurrent slots, then what use is it?

-Tom
=====
On 2013.03.09 10:36, Kris Pister wrote:
Tom - I agree with everything that you wrote in the email below except the first sentence.  Perhaps I'm missing something, but it seems to me that your enumeration of various flows with different requirements fits quite well with the discussion fragment quoted at the end below.  Can you help me understand the differences?

My bias is that we should have
{flow priority (equating to slotframe priority) OR link priority (TX, RX, uni-, multi-)} AND {packet priority}.

ksjp

On 3/8/2013 11:36 PM, Tom Phinney wrote:
I don't know what real-world use this technology is being designed to support, but the discussion fragments quoted at the end below seem to me to be moving away from reality.

In the industrial automation world where I spent most of my career, it makes sense to have different inbound flows for differing purposes. Most sensor devices would have the following flows 1), 2) and 3):
1) a dedicated inbound flow per field device (mote) for periodic deterministic control traffic, where the number of expected hops has to be small when the loop rate is high;
2) a shared inbound flow for usually-infrequent alerts (i.e., device events and process alarms), probably with at least 2-level packet queuing priority, perhaps using CSMA/CA to provide some slight statistical bias in channel access priority based on packet priority;
3) a shared inbound flow for server-to-client responses, typically in response to a program-initiated or operator-initiated action at an engineering workstation in a control room.

Every process actuator (i.e., mote that can directly manipulate the process) will also have another flow 4), analogous to 1) but in the outbound direction, from the control room to the field to transfer the computed actuator setting to the device. Such devices also have inbound flow 1) to transfer actuator status at the same periodic rate.

If communication problems are experienced on flow 1), it is desirable to be able to use one or more of the other flows as a backup mechanism for inbound control traffic, whether full determinism can be maintained or not. Packet queuing priority definitely makes sense here, so that such traffic can leapfrog non-deterministic traffic in the output queue of the originating mote and of any intermediary relaying motes. As mentioned in 2), it may also make sense to use CSMA/CA to provide some slight statistical bias in channel access priority based on packet priority, at least for flows where the transaction template has a non-null CSMA/CA sensing interval before the scheduled Data PDU transmission of the transaction.

The reason for priority on the alerts is that process alarms from class 1 - 2 devices, (or class 0 - 2 if class 0 uses wireless), would usually be configured to be higher priority than those for classes 3 - 5. (Some class 3 devices might also be assigned that higher alarm priority.) Where safety alarms are concerned, such as the "man down" alarm that Herman Storey mentioned on Thursday's call, those alarms would tend to use another flow 5), similar to 2), with a very low expected rate but with dedicated slots to provide a clear virtual channel (to the extent possible). The time constants for such alarms are long enough (seconds) compared to much of the flow 1) traffic that the incremental cost of flow 5), which is so similar to flow 2), probably would not be very much, perhaps 2% of the total network capacity. It is tempting to just have a 3-way CSMA/CA interval for flow 2), but the reality of the CSMA/CA priority assessment mechanism is that it is very weak, given physical modem delays, inter-mote timing skews and the old hidden-node problem. Thus CSMA/CA prioritization does not seem reliable enough to trust for safety purposes.

Of course a system will have other flows, many allocated on a demand basis in response to some occasional need, such as firmware download or captured waveform upload.

The basic timing of continuous control loops it that the total delay from sensing the process to driving the actuator generally has to be less than twice the period of the loop. In such cases the needed control strategy is both relatively simple and relatively stable, However, if the total input-to-output delay exceeds twice the loop period, the control becomes much more difficult and less able to respond to fast transients. Jitter is a problem only when a process value message is delivered in the wrong cycle; it does not matter whether the delivery is at the 10% or 90% point in the proper cycle. Authenticated timestamping of the message, as occurs via the UDP transport nonce in ISA100.11a, provides a means of discarding process value messages that are delivered too late, thus converting those situations to ones of message loss. That is the desired way of handling late delivery, because the control loop is stable under message loss but not when the values are deliberately selectively delayed into the wrong delivery cycle. (Honeywell demonstrated years ago that it was possible to destablize most control loops with such deliberately jittered delivery.)

-Tom
=====

I agree that the term flow is almost as overloaded as link or path. This is not a final term and we will have to come up with a better term. Suggestions?

If I get your point correctly, you are suggesting that there should be no packet priorities, only flows priority. In a TSCH context, this would translate into a slotframe per flow, where flow also incorporates the notion of priority. Correct? This is a very important discussion we started over the phone earlier today, but which we will continue Wednesday in Orlando. I hope you can be there.

Thomas

On Fri, Mar 8, 2013 at 6:55 PM, Michael Richardson <[email-masked]<mailto:[email-masked]>> wrote:
How can a flow have high priority packets and low priority packets?

_______________________________________________
6tsch mailing list
[email-masked]<mailto:[email-masked]>https://www.ietf.org/mailman/listinfo/6tsch

_______________________________________________
6tsch mailing list
[email-masked]<mailto:[email-masked]>https://www.ietf.org/mailman/listinfo/6tsch

---

From: Tom Phinney <[email-masked]>
Date: Sat, 09 Mar 2013 23:38:49 -0700

Shitanshu et al,

I sense that I may be viewing related applications through very
different lenses than the rest of you. The disconnect is
undoubtedly mine, since I am not grounded in IETF terminology the
way that  you all are.

https://tools.ietf.org/id/draft-phinney-roll-rpl-industrial-applicability-02.txt
provides much of the background for the following.

Automation of continuous processes often involves large physical
plants with km of piping, hundreds to thousands of valves and
thousands to tens of thousands of sensors. Much of the high-value
and critical equipment typically is concentrated in areas of about
1 hectare (i.e., 100 m on a side), typically extending 10 m or
more in the air. Other parts are spread over a large area, with
most equipment within 2-3 m of the ground. The largest plant I
ever visited was 10k ha (100 square miles);  the area in which
outdoor communication is required in these plants can be enormous.

At the other extreme, a pharmaceutical plant may contain many
dozen chemical reactors, each with associated piping and valving,
each organized as a column of equipment on a skid that can be
moved and replaced as necessary. Such a structure is suitable for
managing chemical or biological reactions that only work at small
scale, as well as for providing fine-grained traceability of final
product, which latter is required by the health agencies of many
governments.

From an automation perspective these plants are more similar than
dissimilar: each has a set of closed-loop control and monitoring
functions that have specific timeliness requirements, where the
individual flows that represent purpose-specific communications
relationships can be ranked in terms of both criticality and their
acceptable distribution of delivery delay.

In general, criticality varies inversely with

1) the communication class (see 2.1.1 in the above document), and

2) the reporting period. (Note that loop rate is the
inverse of reporting period.)

A third contributing factor is the role of the specific loop in
the plant; within a class at a given loop rate those components
whose failure has a greater consequence tend to be higher on the
scale.

This 3-component ranking does not lead immediately to a linear
priority structure, because control communications is essentially
deadline scheduled rather than prioritized. As noted in the
earlier e-mail, delivery early or late in a given reporting period
makes no difference; it is only missed delivery deadlines that are
consequential. Even then, the control strategies that have evolved
during the last 50 years tolerate a few successive losses
(generally <= 3) after which they mode-shift to a backup regime
that emphasizes plant safety over product.

An example may help. Modern toothpaste is mixed in big batch
reactors. It includes various corrosive and toxic components,
including fluoride. A breakdown in control of the mixing process,
or of history collection due to lost reports, may lead to tonnes
of toxic waste (because governments require uninterrupted history
collection to monitor the contents and safety of such "medical"
products). Although the industry has developed mechanisms to span
and recover from brief to moderate communication outages in
history collection, a failure of the process historian's database
will trigger such a toxic waste disposal problem for the plant
manager.

Getting back to the prior discussion, publish-subscribe
communications may be assigned to flows. When communications
redundancy is employed, as it often is for the highest-ranked
monitoring and control loops, the resulting (usually two) assigned
flows must be disjoint throughout the network, never converging to
any point that would provide a common failure mode until they
reach their final destination, which today is usually a
fault-tolerant server farm.

Source-sink communications, which is used for stateless device
events and stateful process alarms -- the latter go into alarm,
then later return to normal -- has much reduced timeliness
constraints, particularly when there is lots of other alarm
traffic. There is usually a contractual commitment that the first
process alarm that occurs in a period of relatively quiescent
plant operation will be reported within 5 s, often to 4 sigma to 6
sigma confidence. On the other hand, the 70th alarm in a ten
minute period has no reporting timeliness requirements; it simply
should not be lost. As with historizing and control loops, the
industry has developed mechanisms to retrieve process alarm status
even when the reporting messages are lost, so per-message loss is
not a major issue. Application-layer-triggered retries push those
initial alarms persistently; but once the system is flooded with
alarms the delivery requirements change. It is for that reason
that WirelessHART mandated that each 802.15.4 router have only a
single alarm forwarding buffer, causing alarm backlogs to build up
at the originators (where intelligent alarm reordering and
aggregation can occur) rather than in intermediary router queues.

Client-server communications is used primarily to respond to human
or programmatic requests for detailed status of remote devices.
Periodic publish-subscribe communications typically sends only a
single float32 and a coded status byte; when that status byte
indicates problems in the device or with the process, a secondary
retrieval of more detailed information from the device is usually
required. That communication typically requires timeliness
suitable to sustain human interaction; if it takes too long the
operator will attempt to accelerate the process by demanding even
more data.

Because the bandwidth available for such state retrieval is
typically quite low, the centralized side of the automation system
typically caches the data from each such report, thereby being
able to present rapid and relatively reliable information to
centralized automation programs that work with such data. For that
same reason, much of the data that is reported by such
client-server communications is reported by exception (rather than
as a bulky record) when that is permitted.

If the above gives the impression that these systems are complex
and their communications needs not easily ordered into a linear
set, then I have succeeded in providing some background. The most
interesting aspects of these systems is that the plant owners have
substantial incentives to improve their processes and increase
anticipatory maintenance, both of which lead to greater revenue.
The scale of the problems is large, but so are the potential
profits. It is not an accident that many of the world's most
valuable corporations have a significant portion of their profits
generated by such plants. Thus they are more willing than most to
try new technology, and to invest in it heavily if it proves
reliable and improves the bottom line.

Cheers,

-Tom

=====

On 2013.03.09 22:04, Shitanshu Shah (svshah) wrote:

Hi Tom, All,

In general I agree with your assessment for need for packet
priority.

If forwarding service required is similar to all packets for
a given class, packet priority is all what is required for such
services. Like traffic classes in your example for deterministic
class, alerts, server-client response etc..  Treating such class
of traffic at the aggregate level per packet priority (eg. Based
on certain L3 or L2 level code-points) is what is needed for
forwarding(queuing) service.

I also lack to understand, what is the definition of flow, is
it micro-flow that has been eluded before?

For micro-flows belonging to the same service class (eg.
Belonging to alert class), what kind of differentiation needed
for different flows within that class? I would imagine that
queuing is expensive and scarce resources in these systems as
well just like it is for traditional Layer2 switches (at the
least in the hardware). Thus probably very expensive to imagine
as many queues as many set of flows with different priority.

However, a sort of differentiation that can be imagined at
flow level may be for other parameters (but not queuing
behavior). For example, limiting rate of a flow that is fed in
to the queuing system.excessive traffic to be dropped or
re-marked to lower priority code-point. Or during congestion
drop traffic from certain flows over others.

I understand RSVP provides a way to enable integrated queuing
service. At the same time, RSVP also has a way to enable
reservation based on Diffserv. It is later that I am trying to
highlight where packet priority is used for the forwarding
decision, and flow classification may be used to constrain other
parameters like rate.

As I am not so much familiar with Industrial Automation, what
I am not clear if for a given PAN, could there be different set
of flows with different deterministic parameters? If there are
then I can see some impact to queuing discipline and packet
priority itself may not be sufficient. But I am not sure why
there would be an application (PAN) with devices with different
set of deterministic flows. I can also see them causing
conflicts when it comes to channel reservation.

Regards,

Shitanshu

From: Tom Phinney
<[email-masked]>

Reply-To: Tom Phinney
<[email-masked]>

Date: Saturday, March
9, 2013 11:27 AM

To: "[email-masked]"
<[email-masked]>

Subject: Re: [6tsch]
priority and priority

Kris
(et al),

In my post below I was objecting to the conclusion in the
originally quoted interchange,
recolored in red at the end below,
that flow priority suffices. My example flow 2) is one in
which packet priority makes sense WITHIN the flow. Note
also my text
in blue below, highlighting a
similar conclusion for use of flows 2) and/or 3) as backup
channels for flow 1) when flow 1) becomes too unreliable.

My post below was an attempt to point out that queuing
priority AND flow priority need not be disjoint
approaches; in my opinion they can and should both exist
concurrently in systems that deal with real-world
problems, at least in the industrial and process
automation markets. They each provide some functionality
that the other lacks, giving an overall system that is
superior in its performance of critical functions to one
that restricts itself to flow priority, or to queuing
priority, but without supporting both. My post below was
an attempt to sketch a common scenario in that market
where the two priority mechanisms interplay to provide a
superior system.

Perhaps my concept of flow differs from those of the
original two correspondents. If so, I would appreciate
instruction/correction. My scenario below was one where
resources assigned to secondary flows would be used when
the primary flow 1) fails. If others' notions of flow are
not coupled to contracts and resource allocation in
forwarding DL and NL routers,
potentially including slot
prioritization among concurrent slots, then what use is it?

-Tom

=====

Tom - I agree with everything that you wrote
in the email below except the first sentence.  Perhaps I'm
missing something, but it seems to me that your
enumeration of various flows with different requirements
fits quite well with the discussion fragment quoted at the
end below.  Can you help me understand the differences?

{flow priority (equating to slotframe priority) OR link
priority (TX, RX, uni-, multi-)} AND {packet priority}.

ksjp

On 3/8/2013 11:36 PM, Tom
Phinney wrote:

I don't know what real-world use this
technology is being designed to support, but the
discussion fragments quoted at the end below seem to me
to be moving away from reality.

In the industrial automation world where I spent most of
my career, it makes sense to have different inbound
flows for differing purposes. Most sensor devices would
have the following flows 1), 2) and 3):

1) a dedicated inbound flow per field device (mote) for
periodic deterministic control traffic, where the number
of expected hops has to be small when the loop rate is
high;

2) a shared inbound flow for usually-infrequent alerts
(i.e., device events and process alarms), probably with
at least 2-level packet queuing priority, perhaps using
CSMA/CA to provide some slight statistical bias in
channel access priority based on packet priority;

3) a shared inbound flow for server-to-client responses,
typically in response to a program-initiated or
operator-initiated action at an engineering workstation
in a control room.

Every process actuator (i.e., mote that can directly
manipulate the process) will also have another flow 4),
analogous to 1) but in the outbound direction, from the
control room to the field to transfer the computed
actuator setting to the device. Such devices also have
inbound flow 1) to transfer actuator status at the same
periodic rate.

If communication problems are experienced on flow 1), it is desirable to be able to use one
or more of the other flows as a backup mechanism for
inbound control traffic, whether full
determinism can be maintained or not.
Packet queuing priority definitely
makes sense here, so that such traffic can leapfrog
non-deterministic traffic in the output queue of the
originating mote and of any intermediary relaying
motes. As mentioned in 2), it may also make
sense to use CSMA/CA to provide some slight statistical
bias in channel access priority based on packet
priority, at least for flows where the transaction
template has a non-null CSMA/CA sensing interval before
the scheduled Data PDU transmission of the transaction.

The reason for priority on the alerts is that process
alarms from class 1 - 2 devices, (or class 0 - 2 if
class 0 uses wireless), would usually be configured to
be higher priority than those for classes 3 - 5. (Some
class 3 devices might also be assigned that higher alarm
priority.) Where safety alarms are concerned, such as
the "man down" alarm that Herman Storey mentioned on
Thursday's call, those alarms would tend to use another
flow 5), similar to 2), with a very low expected rate
but with dedicated slots to provide a clear virtual
channel (to the extent possible). The time constants for
such alarms are long enough (seconds) compared to much
of the flow 1) traffic that the incremental cost of flow
5), which is so similar to flow 2), probably would not
be very much, perhaps 2% of the total network capacity.
It is tempting to just have a 3-way CSMA/CA interval for
flow 2), but the reality of the CSMA/CA priority
assessment mechanism is that it is very weak, given
physical modem delays, inter-mote timing skews and the
old hidden-node problem. Thus CSMA/CA prioritization
does not seem reliable enough to trust for safety
purposes.

Of course a system will have other flows, many allocated
on a demand basis in response to some occasional need,
such as firmware download or captured waveform upload.

The basic timing of continuous control loops it that the
total delay from sensing the process to driving the
actuator generally has to be less than twice the period
of the loop. In such cases the needed control strategy
is both relatively simple and relatively stable,
However, if the total input-to-output delay exceeds
twice the loop period, the control becomes much more
difficult and less able to respond to fast transients.
Jitter is a problem only when a process value message is
delivered in the wrong cycle; it does not matter whether
the delivery is at the 10% or 90% point in the proper
cycle. Authenticated timestamping of the message, as
occurs via the UDP transport nonce in ISA100.11a,
provides a means of discarding process value messages
that are delivered too late, thus converting those
situations to ones of message loss. That is the desired
way of handling late delivery, because the control loop
is stable under message loss but not when the values are
deliberately selectively delayed into the wrong delivery
cycle. (Honeywell demonstrated years ago that it was
possible to destablize most control loops with such
deliberately jittered delivery.)

-Tom

=====

Michael,

I agree that the term flow is almost as
overloaded as link or path. This is not a final term
and we will have to come up with a better term.
Suggestions?

If I get your point correctly, you are suggesting
that
there should be no packet priorities, only flows
priority. In a TSCH context, this would
translate into a slotframe per flow, where flow also
incorporates the notion of priority. Correct? This
is a very important discussion we started over the
phone earlier today, but which we will continue
Wednesday in Orlando. I hope you can be there.

Thomas

On Fri, Mar 8, 2013 at 6:55
PM, Michael Richardson
<[email-masked]>
wrote:

How can a flow have high
priority packets and low priority packets?

Why are these two types of packets using the
same DODAG?

_______________________________________________
6tsch mailing list
[email-masked]://www.ietf.org/mailman/listinfo/6tsch

_______________________________________________
6tsch mailing list
[email-masked]://www.ietf.org/mailman/listinfo/6tsch

---

From: Kris Pister <[email-masked]>
Date: Sun, 10 Mar 2013 13:25:41 -0700

Shitanshu -
 > As I am not so much familiar with Industrial Automation, what I am not
 > clear if for a given PAN, could there be different set of flows with 
different
 > deterministic parameters?
    I don't think that the things that Tom is writing about are specific 
to industrial automation.  Most WSN applications have similar kinds of 
flows.  Building automation, smart grid, smart home, etc. all have most 
if not all of the various flows and requirements enumerated in rfc5637.

For example, light switches, burglar alarms, and thermostats have very 
different deterministic requirements.

ksjp

On 3/9/2013 9:04 PM, Shitanshu Shah (svshah) wrote:
> If forwarding service required is similar to all packets for a given 
> class, packet priority is all what is required for such services. 
> Like traffic classes in your example for deterministic class, alerts, 
> server-client response etc..  Treating such class of traffic at the 
> aggregate level per packet priority (eg. Based on certain L3 or L2 
> level code-points) is what is needed for forwarding(queuing) service.
>
> For micro-flows belonging to the same service class (eg. Belonging to 
> alert class), what kind of differentiation needed for different flows 
> within that class? I would imagine that queuing is expensive and 
> scarce resources in these systems as well just like it is for 
> traditional Layer2 switches (at the least in the hardware). Thus 
> probably very expensive to imagine as many queues as many set of flows 
> with different priority.
>
> However, a sort of differentiation that can be imagined at flow level 
> may be for other parameters (but not queuing behavior). For example, 
> limiting rate of a flow that is fed in to the queuing system.excessive 
> traffic to be dropped or re-marked to lower priority code-point. Or 
> during congestion drop traffic from certain flows over others.
>
> I understand RSVP provides a way to enable integrated queuing service. 
> At the same time, RSVP also has a way to enable reservation based on 
> Diffserv. It is later that I am trying to highlight where packet 
> priority is used for the forwarding decision, and flow classification 
> may be used to constrain other parameters like rate.
>
> As I am not so much familiar with Industrial Automation, what I am not 
> clear if for a given PAN, could there be different set of flows with 
> different deterministic parameters? If there are then I can see some 
> impact to queuing discipline and packet priority itself may not be 
> sufficient. But I am not sure why there would be an application (PAN) 
> with devices with different set of deterministic flows. I can also see 
> them causing conflicts when it comes to channel reservation.
>
> Regards,
> Shitanshu
>
>
> From: Tom Phinney <[email-masked] <mailto:[email-masked]>>
> Reply-To: Tom Phinney <[email-masked] <mailto:[email-masked]>>
> To: "[email-masked] <mailto:[email-masked]>" <[email-masked] 
> <mailto:[email-masked]>>

---

From: "Shitanshu Shah (svshah)" <[email-masked]>
Date: Mon, 11 Mar 2013 00:51:03 +0000

Hi Tom,

If the above gives the impression that these systems are complex and their communications needs not easily ordered into a linear set, then I have succeeded in providing some background

##svshah, My response is not highlighting it to be strict linear set priority service. Perhaps my earlier response did not do a good job. Infact Diffserv code-points (as well any L2 code-points) and their PHB definition do not define them a strict ordered priority service.

Thanks
Shitanshu

Date: Sunday, March 10, 2013 1:38 AM

Shitanshu et al,

I sense that I may be viewing related applications through very different lenses than the rest of you. The disconnect is undoubtedly mine, since I am not grounded in IETF terminology the way that  you all are.

https://tools.ietf.org/id/draft-phinney-roll-rpl-industrial-applicability-02.txt provides much of the background for the following.

Automation of continuous processes often involves large physical plants with km of piping, hundreds to thousands of valves and thousands to tens of thousands of sensors. Much of the high-value and critical equipment typically is concentrated in areas of about 1 hectare (i.e., 100 m on a side), typically extending 10 m or more in the air. Other parts are spread over a large area, with most equipment within 2-3 m of the ground. The largest plant I ever visited was 10k ha (100 square miles);  the area in which outdoor communication is required in these plants can be enormous.

At the other extreme, a pharmaceutical plant may contain many dozen chemical reactors, each with associated piping and valving, each organized as a column of equipment on a skid that can be moved and replaced as necessary. Such a structure is suitable for managing chemical or biological reactions that only work at small scale, as well as for providing fine-grained traceability of final product, which latter is required by the health agencies of many governments.

>From an automation perspective these plants are more similar than dissimilar: each has a set of closed-loop control and monitoring functions that have specific timeliness requirements, where the individual flows that represent purpose-specific communications relationships can be ranked in terms of both criticality and their acceptable distribution of delivery delay.

In general, criticality varies inversely with
 1) the communication class (see 2.1.1 in the above document), and
 2) the reporting period. (Note that loop rate is the inverse of reporting period.)

A third contributing factor is the role of the specific loop in the plant; within a class at a given loop rate those components whose failure has a greater consequence tend to be higher on the scale.

This 3-component ranking does not lead immediately to a linear priority structure, because control communications is essentially deadline scheduled rather than prioritized. As noted in the earlier e-mail, delivery early or late in a given reporting period makes no difference; it is only missed delivery deadlines that are consequential. Even then, the control strategies that have evolved during the last 50 years tolerate a few successive losses (generally <= 3) after which they mode-shift to a backup regime that emphasizes plant safety over product.

An example may help. Modern toothpaste is mixed in big batch reactors. It includes various corrosive and toxic components, including fluoride. A breakdown in control of the mixing process, or of history collection due to lost reports, may lead to tonnes of toxic waste (because governments require uninterrupted history collection to monitor the contents and safety of such "medical" products). Although the industry has developed mechanisms to span and recover from brief to moderate communication outages in history collection, a failure of the process historian's database will trigger such a toxic waste disposal problem for the plant manager.

Getting back to the prior discussion, publish-subscribe communications may be assigned to flows. When communications redundancy is employed, as it often is for the highest-ranked monitoring and control loops, the resulting (usually two) assigned flows must be disjoint throughout the network, never converging to any point that would provide a common failure mode until they reach their final destination, which today is usually a fault-tolerant server farm.

Source-sink communications, which is used for stateless device events and stateful process alarms -- the latter go into alarm, then later return to normal -- has much reduced timeliness constraints, particularly when there is lots of other alarm traffic. There is usually a contractual commitment that the first process alarm that occurs in a period of relatively quiescent plant operation will be reported within 5 s, often to 4 sigma to 6 sigma confidence. On the other hand, the 70th alarm in a ten minute period has no reporting timeliness requirements; it simply should not be lost. As with historizing and control loops, the industry has developed mechanisms to retrieve process alarm status even when the reporting messages are lost, so per-message loss is not a major issue. Application-layer-triggered retries push those initial alarms persistently; but once the system is flooded with alarms the delivery requirements change. It is for that reason that WirelessHART mandated that each 802.15.4 router have only a single alarm forwarding buffer, causing alarm backlogs to build up at the originators (where intelligent alarm reordering and aggregation can occur) rather than in intermediary router queues.

Client-server communications is used primarily to respond to human or programmatic requests for detailed status of remote devices. Periodic publish-subscribe communications typically sends only a single float32 and a coded status byte; when that status byte indicates problems in the device or with the process, a secondary retrieval of more detailed information from the device is usually required. That communication typically requires timeliness suitable to sustain human interaction; if it takes too long the operator will attempt to accelerate the process by demanding even more data.

Because the bandwidth available for such state retrieval is typically quite low, the centralized side of the automation system typically caches the data from each such report, thereby being able to present rapid and relatively reliable information to centralized automation programs that work with such data. For that same reason, much of the data that is reported by such client-server communications is reported by exception (rather than as a bulky record) when that is permitted.

If the above gives the impression that these systems are complex and their communications needs not easily ordered into a linear set, then I have succeeded in providing some background. The most interesting aspects of these systems is that the plant owners have substantial incentives to improve their processes and increase anticipatory maintenance, both of which lead to greater revenue. The scale of the problems is large, but so are the potential profits. It is not an accident that many of the world's most valuable corporations have a significant portion of their profits generated by such plants. Thus they are more willing than most to try new technology, and to invest in it heavily if it proves reliable and improves the bottom line.

Cheers,
-Tom
=====

Hi Tom, All,

Regards,
Shitanshu

Kris (et al),

-Tom
=====

ksjp

-Tom
=====

Thomas

---

From: "Shitanshu Shah (svshah)" <[email-masked]>
Date: Mon, 11 Mar 2013 01:09:59 +0000

Hi Kris,

> For example, light switches, burglar alarms, and thermostats have very different deterministic requirements.

Interestingly enough all of them may co-exist in one WSN network. If they have different deterministic requirements, eg. hypothetically taking an example timely delivery (I take deterministic = jitter sensitive) every 2 seconds, 3 seconds and 5 seconds respectively. If they follow the same path, what would be an expectation when they conflict with each other for a fixed slot delivery? For example, light switches and burglar alarms may conflict at 6th second. Light switches and thermostats at 10th second. Or am I on some tangent here? If I am completely off the track then feel free to ignore. meeting with folks here at the IETF will make me straight.

btw, do control signals for light switches, thermostats etc. are jitter sensitive (and so fall under deterministic service requirements?)

Thanks
Shitanshu

From: Kris Pister <[email-masked]<mailto:[email-masked]>>
Date: Sunday, March 10, 2013 4:25 PM
To: Shitanshu Shah <[email-masked]<mailto:[email-masked]>>
Cc: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, Tom Phinney <[email-masked]<mailto:[email-masked]>>
Subject: Re: [6tsch] priority and priority

Shitanshu -
> clear if for a given PAN, could there be different set of flows with different
   I don't think that the things that Tom is writing about are specific to industrial automation.  Most WSN applications have similar kinds of flows.  Building automation, smart grid, smart home, etc. all have most if not all of the various flows and requirements enumerated in rfc5637.

For example, light switches, burglar alarms, and thermostats have very different deterministic requirements.

ksjp

Hi Tom, All,

Regards,
Shitanshu

Kris (et al),

-Tom
=====

ksjp

-Tom
=====

Thomas

---

From: Kris Pister <[email-masked]>
Date: Mon, 11 Mar 2013 10:47:02 -0700

Shitanshu - I think that you're on the right track.  Here's how I look 
at the WSN world:

For energy efficiency in WSN, we'd prefer to never transmit anything.  
Unfortunately for battery life, users set requirements on data reception 
rate, latency, and reliability.
The goal of WSN is to meet these L4 user QoS requirements via L3 routes 
on L2 waking schedules, while minimizing global power and traffic, 
subject to node-local power and energy constraints.
 From that perspective, *every* flow in the network is jitter sensitive 
and deterministic.  If not, then there are resources being wasted.  When 
I ask people about their latency and reliability requirements, I often 
hear "oh, I don't care much about latency and reliability", and I say 
"Great!  I'll give you one packet per month with 10% probability".

Once you boil the data flows down to a set of {rate, latency, 
reliability} tuples, then you can start to allocate L3 and L2 resources 
to achieve whatever level of safety margin is appropriate.  In a network 
with no scheduled cell collisions (one of many ways to do it) It's the 
shared safety margin in L2 provisioning where your questions about 
priorities get interesting.

ksjp

On 3/10/2013 6:09 PM, Shitanshu Shah (svshah) wrote:
>
> Hi Kris,
>
> > For example, light switches, burglar alarms, and thermostats have 
> very different deterministic requirements.
>
> Interestingly enough all of them may co-exist in one WSN network. If 
> they have different deterministic requirements, eg. hypothetically 
> taking an example timely delivery (I take deterministic = jitter 
> sensitive) every 2 seconds, 3 seconds and 5 seconds respectively. If 
> they follow the same path, what would be an expectation when they 
> conflict with each other for a fixed slot delivery? For example, light 
> switches and burglar alarms may conflict at 6th second. Light switches 
> and thermostats at 10th second. Or am I on some tangent here? If I am 
> completely off the track then feel free to ignore. meeting with folks 
> here at the IETF will make me straight.
>
> btw, do control signals for light switches, thermostats etc. are 
> jitter sensitive (and so fall under deterministic service requirements?)
>
> Thanks
> Shitanshu
>
>
> To: Shitanshu Shah <[email-masked] <mailto:[email-masked]>>
> Cc: "[email-masked] <mailto:[email-masked]>" <[email-masked] 
> <mailto:[email-masked]>>, Tom Phinney <[email-masked] 
> <mailto:[email-masked]>>
>    I don't think that the things that Tom is writing about are 
> specific to industrial automation.  Most WSN applications have similar 
> kinds of flows.  Building automation, smart grid, smart home, etc. all 
> have most if not all of the various flows and requirements enumerated 
> in rfc5637.
>
>> probably very expensive to imagine as many queues as many set of 
>> flows with different priority.
>>
>> limiting rate of a flow that is fed in to the queuing 
>> system.excessive traffic to be dropped or re-marked to lower priority 
>> code-point. Or during congestion drop traffic from certain flows over 
>> others.
>>
>> service. At the same time, RSVP also has a way to enable reservation 
>> based on Diffserv. It is later that I am trying to highlight where 
>> packet priority is used for the forwarding decision, and flow 
>> classification may be used to constrain other parameters like rate.
>>
>> As I am not so much familiar with Industrial Automation, what I am 
>> not clear if for a given PAN, could there be different set of flows 
>> with different deterministic parameters? If there are then I can see 
>> some impact to queuing discipline and packet priority itself may not 
>> be sufficient. But I am not sure why there would be an application 
>> (PAN) with devices with different set of deterministic flows. I can 
>> also see them causing conflicts when it comes to channel reservation.
>>
>> Regards,
>> Shitanshu
>>
>>
>> real-world problems, at least in the industrial and process 
>> flow priority, or to queuing priority, but without supporting both. 
>> My post below was an attempt to sketch a common scenario in that 
>> market where the two priority mechanisms interplay to provide a 
>> superior system.
>>
>>>> device events and process alarms), probably with at least 2-level 
>>>> packet queuing priority, perhaps using CSMA/CA to provide some 
>>>> slight statistical bias in channel access priority based on packet 
>>>> priority;
>>>> call, those alarms would tend to use another flow 5), similar to 
>>>> 2), with a very low expected rate but with dedicated slots to 
>>>> provide a clear virtual channel (to the extent possible). The time 
>>>> constants for such alarms are long enough (seconds) compared to 
>>>> much of the flow 1) traffic that the incremental cost of flow 5), 
>>>> which is so similar to flow 2), probably would not be very much, 
>>>> perhaps 2% of the total network capacity. It is tempting to just 
>>>> have a 3-way CSMA/CA interval for flow 2), but the reality of the 
>>>> CSMA/CA priority assessment mechanism is that it is very weak, 
>>>> given physical modem delays, inter-mote timing skews and the old 
>>>> demand basis in response to some occasional need, such as firmware 
>>>> download or captured waveform upload.
>>>>
>>>> has to be less than twice the period of the loop. In such cases the 
>>>> needed control strategy is both relatively simple and relatively 
>>>> stable, However, if the total input-to-output delay exceeds twice 
>>>> the loop period, the control becomes much more difficult and less 
>>>> able to respond to fast transients. Jitter is a problem only when a 
>>>> process value message is delivered in the wrong cycle; it does not 
>>>> matter whether the delivery is at the 10% or 90% point in the 
>>>> proper cycle. Authenticated timestamping of the message, as occurs 
>>>> via the UDP transport nonce in ISA100.11a, provides a means of 
>>>> discarding process value messages that are delivered too late, thus 
>>>> converting those situations to ones of message loss. That is the 
>>>> desired way of handling late delivery, because the control loop is 
>>>> stable under message loss but not when the values are deliberately 
>>>>> path. This is not a final term and we will have to come up with a 
>>>>> better term. Suggestions?
>>>>>
>>>>> flow also incorporates the notion of priority. Correct? This is a 
>>>>> very important discussion we started over the phone earlier today, 
>>>>> but which we will continue Wednesday in Orlando. I hope you can be 
>>>>> there.
>>>>>
>>>>> Thomas
>>>>>

---

From: Michael Richardson <[email-masked]>
Date: Mon, 11 Mar 2013 15:43:10 -0400

>>>>> "Thomas" == Thomas Watteyne <[email-masked]> writes:
    Thomas> I agree that the term flow is almost as overloaded as link
    Thomas> or path. This is not a final term and we will have to come
    Thomas> up with a better term.  Suggestions?

I don't have a suggestion as to a term.
The thing that I do when I find out that all the normal terms are
overloaded is to either:   
  a) use the same term from another language
  b) make up a new term, either nonsense or ideally a porte-manteau.

I believe that terminology is important and getting the right words in
place makes the discussion go much smoother.  Overloading words is
bad...  It's good that we talk about frames, fragments, packets and
segments with much clarity.

let me reply seperately for the other part, changing the subject line.

-- 
Michael Richardson
-on the road-

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Mon, 11 Mar 2013 19:48:39 +0000

I agree, flow is misleading.

We have the concept of a track. Why not say track priority?

Pascal

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Michael Richardson
Sent: lundi 11 mars 2013 15:43
To: IETF 6TSCH
Subject: [6tsch] what to call "flow" with different priorities

I believe that terminology is important and getting the right words in place makes the discussion go much smoother.  Overloading words is bad...  It's good that we talk about frames, fragments, packets and segments with much clarity.

--
Michael Richardson
-on the road-

---

From: Michael Richardson <[email-masked]>
Date: Mon, 11 Mar 2013 16:49:01 -0400

>>>>> "Thomas" == Thomas Watteyne <[email-masked]> writes:
    Thomas> If I get your point correctly, you are suggesting that there
    Thomas> should be no packet priorities, only flows priority. In a
    Thomas> TSCH context, this would translate into a slotframe per
    Thomas> flow, where flow also incorporates the notion of
    Thomas> priority. Correct? This is a very important discussion we
    Thomas> started over the phone earlier today, but which we will
    Thomas> continue Wednesday in Orlando. I hope you can be there.

I think that we need to start from goals, and talk about what methods
will be used to configure and/or provision them.  Let's not confuse
mechanism (DSCP, slotframes) from goals.

My understanding is that we wish to use a non-protected ("CAP" is what I
understand to be the 802.15.4 term) in order to bring up the network,
determine what the parent relationship could be, and then use that
control channel to bootstrap the rest of the system.  This will happen
with RPL creating an initial DODAG.

My further understanding is that we will then allocate/provision
slotframes for communication between adjacent nodes.  How big these
slotframes are is out of scope for the moment(%).

Once these slotframes are created, then they will be expressed upwards
to the RPL layer as links with specific ETXs, possibly using a to be
defined new set of (time) based metrics.  Based upon traffic constraints
expressed by the application(s), new DODAGs will be created using the
new (presumably lower latencies and higher bandwidths) adjacencies.

Now we have a problem: the various DODAGs may well interact poorly if
their total of their requirements exceed the resources available on that
slotframe.  If we make these as all hard reservation and we therefore do
not permit oversubscription, we are good.  If we loose a parent, then
each DODAG has to go and think (probably in a priority reservation
order), where to get an equivalent path via another parent.    
The parent selection algorith will have seen a particular ETX, but that
may no longer be available after higher priority DODAGs have claimed
pieces of them.

We do not need in-band per-packet markings, at least for storing mode
DODAGs, as we will have the RPLinstanceID already marked in-band.  

I don't think that one needs or wants a slotframe per DODAG (although it
could degenerate to that), nor do I think that this is useful, as fact
different packets (particularly downward ones!) go out actually in
different ways. 

So we wind up with multiple queues (one per DODAG) per slotframe, which
are serviced by the slotframes in priority order. 

(%)- now above I said that how big the slotframes would be decided out
of scope, but in fact, I get the impression that we actually want to
figure out the DODAG structure, and then once we know what adjacencies
are possible, that we then decide how big a slotframe to allocate.

-- 
Michael Richardson
-on the road-

---

From: Kris Pister <[email-masked]>
Date: Mon, 11 Mar 2013 14:11:55 -0700

On 3/11/2013 1:49 PM, Michael Richardson wrote:
> My understanding is that we wish to use a non-protected ("CAP" is what 
> I understand to be the 802.15.4 term)
Just to make sure we're on the same page:
CAP = "contention access period" has a specific meaning in 15.4. It's 
the CSMA period after a beacon in the original standard, also used by 
the DSME and LE MACs in 4e.  If CAP were not already defined for these 
15.4 modes, then using a slotted aloha (contention-based) mechanism for 
initial communication might reasonably be called CAP, but given that the 
term is already defined elsewhere we should avoid it here.

> the various DODAGs may well interact poorly if their total of their 
> requirements exceed the resources available on that slotframe.
Hmm.  I'm missing something.

> I don't think that one needs or wants a slotframe per DODAG (although 
> it could degenerate to that), nor do I think that this is useful, as 
> fact different packets (particularly downward ones!) go out actually 
> in different ways. 
Can you say more about this?  I was thinking that we want at least one 
slotframe per DODAG.

> are possible, that we then decide how big a slotframe to allocate.
That sounds good, and is something that gets done dynamically, yes?

ksjp

---

From: "Timothy J. Salo" <[email-masked]>
Date: Mon, 11 Mar 2013 20:00:54 -0500

>    b) make up a new term, either nonsense or ideally a porte-manteau.

Often, I will simply prepend an adjective to a word that is appropriate,
but overloaded.  For example, I might write something like "6tsch flow"
to indicate that I am referring to something that is similar to,
and yet distinct from, other uses of the term.  No, I not proposing that
the "6tsch flow" be used; I am merely suggesting that this approach be
considered.

In my view, this strategy has the advantage that the document often
reads well compared to, for example, the use of a lot of acronyms,
strings of characters that might resemble several words smushed
together, foreign words or phrases[1], or non-words.

I particularly like the use of qualified nouns that are suggestive of
their meaning when I am researching a specific topic.  Often, I will
read a page or two of a document, rather than the whole document from
front-to-back.  In these cases, I want to quickly determine whether
these particular pages are relevant to my search, without having to
penetrate document-specific terms, or terms or usage that are
unique to a narrow topic.

> bad...

I agree.  I also believe that writers should be kind to their readers.
In my view, this includes writing in English, and avoiding excessive use
of acronyms, odd character strings that are unique to the document, and
other non-words.

[1] quidquid Latine dictum sit, altum sonatur
     (Whatever is said in Latin sounds profound )

-tjs

---

From: Michael Richardson <[email-masked]>
Date: Tue, 12 Mar 2013 11:55:42 -0400

>>>>> "Michael" == Michael Richardson <[email-masked]> writes:
    Michael> bad...  It's good that we talk about frames, fragments,
    Michael> packets and 
    Michael> segments with much clarity.

BTW: Ralph has suggested that the things that 6lowPAN produces are "fraglets"

-- 
Michael Richardson
-on the road-

---

From: Michael Richardson <[email-masked]>
Date: Tue, 12 Mar 2013 12:05:56 -0400

>>>>> "Kris" == Kris Pister <[email-masked]> writes:
    >> understand to be the 802.15.4 term)

    Kris> Just to make sure we're on the same page:
    Kris> CAP = "contention access period" has a specific meaning in
    Kris> 15.4. It's the 
    Kris> CSMA period after a beacon in the original standard, also used
    Kris> by the DSME 

yes, that's what I meant. I believe that 4e can still have a Contention
Access Period?

    Kris> and LE MACs in 4e.  If CAP were not already defined for these 15.4 modes,
    Kris> then using a slotted aloha (contention-based) mechanism for initial
    Kris> communication might reasonably be called CAP, but given that
    Kris> the term is 
    Kris> already defined elsewhere we should avoid it here.

So, does 4e define "sloted aloha"? I'm halfway through it that document.

    >> different packets (particularly downward ones!) go out actually in

    Kris> Can you say more about this?  I was thinking that we want at
    Kris> least one 
    Kris> slotframe per DODAG.

But, my understanding is that a slotframe is used to communicate from
mote A to mote B (at time T, on frequency l).

A node in the DODAG will in general have one or more parents, and one or
more children.   So at a minimum you need a slotframe for each adjacancy
that the DODAG has created.  If there are multiple applications using
the LLN, then you may have multiple DODAGs (with different parameters),
so if each DODAG needs a set of slotframes, then I think we may well
easily run out of slotframes.

That's why I think that each DODAG gets a reservation on one or more
slotframes.

    >> (%)- now above I said that how big the slotframes would be
    >> decided out of 
    >> scope, but in fact, I get the impression that we actually want to figure
    >> out the DODAG structure, and then once we know what adjacencies are
    >> possible, that we then decide how big a slotframe to allocate.

    Kris> That sounds good, and is something that gets done dynamically,
    Kris> yes? 

My impression is that the information goes up into a PCE, which crunches
a bunch of data, and produces a schedule.  That work doesn't happen
frequently, and in may in fact occur once only after the network is initially
provisioned.

-- 
Michael Richardson
-on the road-

---

From: Tom Phinney <[email-masked]>
Date: Tue, 12 Mar 2013 11:37:48 -0700

My vote is for "fraglets" and "fragleting" and
"fragletation".

Fragmentation is the IETF term for segmentation at the network
layer, presumably distinguishing it from segmentation that occurs
above the communication layers (e.g., in the application layer).
Since the inverse of segmentation, which is recombination or
reassembly, is already used both for OSI segmentation and IETF
fragmentation, it might as well be used for 6LoWPAN fragletation
as well.

-Tom

====

On 2013.03.12 08:55, Michael Richardson wrote:

---

From: Xavier Vilajosana <[email-masked]>
Date: Tue, 12 Mar 2013 13:53:12 -0700

Hi Michael,

just to clarify.

a slotframe is a collection of slots (namely "A MAC-level abstraction that is internal to the node and contains a series of timeslots of equal length and priority."

What you refer to is a Cell "A single element in the TSCH schedule, identified by a  slotOffset and a channelOffset value)

>> easily run out of slotframes.

A node can have a scheduled cell to each of its parents (maybe only to those that you want to communicate to). If multiple DODAGs co-exists, and the routes they have use same parents, there is no need to have more cells on the schedule as long as the required bandwidth is met.

does make sense to you?

X

---

From: Michael Richardson <[email-masked]>
Date: Wed, 13 Mar 2013 22:52:34 -0400

>>>>> "Xavier" == Xavier Vilajosana <[email-masked]> writes:

    Xavier> a slotframe is a collection of slots (namely "A MAC-level
    Xavier> abstraction that is internal to the node and contains a
    Xavier> series of timeslots of equal length and priority."

Do slotframes go to one destination or many?  (I'm not including
multicast here)

-- 
Michael Richardson
-on the road-

---

From: Xavier Vilajosana <[email-masked]>
Date: Thu, 14 Mar 2013 07:54:58 -0700

Hi Michael,

> Do slotframes go to one destination or many? (I'm not including 
If you look to a particular node, its slotframe can have several entries 
already allocated and pointing to one or more destinations (so 
destinations can be many and this many destinations for a particular 
node are its neighbors (1hop))

However, when you look at the slotframe from the point of view of the 
network (i.e overlapping all slotframes of all nodes in the network) you 
have a map of what it is happening in general to the network, the 
resulting slotframe (overlap of all nodes slotframe) can have some cells 
(the same one) that are allocated to different nodes that cannot heard 
each other and thus can transmit simultaneously. However this 
overlapping tends to be sparse as the slotframe size is big compared to 
the required traffic.
The allocation of the cells to the slotframe is done by a PCE or by a 
distributed approach at each node.

hope this is more clear now.

regards
Xavi

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Thu, 14 Mar 2013 18:25:20 +0000

Hi Michael:

A particular cell in the slotframe may be reserved for a particular point to point flow (e.g. a control loop). In that case, the cell is equivalent to an MPLS label in that the sequence of cells is programmed (we call that a track) along the (multihop) path between Alice and Bob and only leads to Bob. Other cells are used for QoS based traffic and thus the utilization depends on statistically muxed IP traffic between multiple pairs of endpoint that is routed over the link between the sender and receiver routers on that cell.

Was that your question?

Pascal

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Michael Richardson
Sent: mercredi 13 mars 2013 22:53
To: Xavier Vilajosana
Cc: [email-masked]
Subject: Re: [6tsch] how to handle different classes of traffic

Do slotframes go to one destination or many?  (I'm not including multicast here)

--
Michael Richardson
-on the road-

---

From: Michael Richardson <[email-masked]>
Date: Mon, 18 Mar 2013 09:18:37 -0400

>>>>> "pthubert" == pthubert  <Pascal> writes:
    pthubert> A particular cell in the slotframe may be reserved for a
    pthubert> particular point to point flow (e.g. a control loop). In
    pthubert> that case, the cell is equivalent to an MPLS label in that
    pthubert> the sequence of cells is programmed (we call that a track)
    pthubert> along the (multihop) path between Alice and Bob and only
    pthubert> leads to Bob. Other cells are used for QoS based traffic

Is this inherent in 6tsch, or is this the desired result of the PCE
calculation that would take into account layer-3 forwarding?

-- 
Michael Richardson <[email-masked]>, Sandelman Software Works

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Mon, 18 Mar 2013 14:29:07 +0000

Hello Michael

At the end of the day it's more a matter of use cases than technology since both centralized and distributed could do statistical mux.

In practice, though, it looks like it's the latter; for all I know, the PCE related use cases only involve deterministic tracks such that each new flow will get its own new track of newly allocated cells.

6TSCH adds RPL routing to scale a great many of less critical flows. This is where we expect the traditional QoS with DSCPs, RED, etc...

Cheers,

Pascal

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Michael Richardson
Sent: lundi 18 mars 2013 14:19
To: Pascal Thubert (pthubert)
Cc: Xavier Vilajosana; [email-masked]
Subject: Re: [6tsch] how to handle different classes of traffic

Is this inherent in 6tsch, or is this the desired result of the PCE calculation that would take into account layer-3 forwarding?

---

From: Thomas Watteyne <[email-masked]>
Date: Mon, 18 Mar 2013 09:12:56 -0700

Michael,
Just to quickly complement, TSCH does not include the concept of labeling
cells for label-based switching, this is something we're contemplating for
6TSCH.
Thomas

On Mon, Mar 18, 2013 at 7:29 AM, Pascal Thubert (pthubert) <
[email-masked]> wrote:

> Hello Michael
>
> At the end of the day it's more a matter of use cases than technology
> since both centralized and distributed could do statistical mux.
>
> In practice, though, it looks like it's the latter; for all I know, the
> PCE related use cases only involve deterministic tracks such that each new
> flow will get its own new track of newly allocated cells.
>
> 6TSCH adds RPL routing to scale a great many of less critical flows. This
> is where we expect the traditional QoS with DSCPs, RED, etc...
>
> Cheers,
>
> Pascal
>
>