Subject: RE: QoS in Shared Media

From: Juha Heinanen <[email-masked]>
Date: Mon, 30 Jul 2001 18:53:26 +0300 (EEST)

jay,

i'm reading old emails and found your dpr/rpr/diffserv comment below to
which i have not had time to reply.  unfortunately i have to disagree
with you regarding dpt/rpr supporting diffserv.  the problem is their
node based fairness concept.

lets say that i have a 100 mbps rpr ring of four nodes a-b-c-d-a, there
may be 25 mbps of yellow and red af1 packets from both a to d and b to d
plus 70 mbps of green af1 packets from c to a.  in c, yellow and red
packets from a and b to d should not get 1/2 (50 mbps) of the bandwidth
causing 20 mbps of green af1 packets originating from c to be dropped.

-- juha
------------------------------------------------
Jay Wang writes:
 > 
 > Dunno what is the problem. To support diffserv, the underneath
 > transport (MAC layer and lower), does not have to have a
 > sophisticated traffic management matching diffserv PHBs. The fact
 > that cisco's DPT (or others RPR version for that matter), a MAC
 > layer technology, that supports only two transmit queueing priorities,
 > hence is not a disqualifier for supporting diffserv on the higher level,
 > where you suggested otherwise. In a share-media environment, from
 > diffserv's perspective, what we need from below is a link layer
 > that may ensure a quantifiable and stable guaranteed throughput for the
 > target nodes, regardless of the other sharing nodes. This is what is
 > missing in the original IEEE-802-style LAN. The DPT, on the other hand,
 > assures total_ring-BandWidth/node_num for each node. Given that, the
 > business of diffserv traffic management, e.g., BA classification,
 > policing, color-based RED dropping, PHB scheduling, and so on and so forth,
 > can be conducted as they should have been at the connected nodes.
 > 
 > - jay
 > 
 > 
 > -----Original Message-----
 > From: Juha Heinanen [mailto:[email-masked]]
 > Sent: Friday, May 04, 2001 10:52 PM
 > To: Jay Wang
 > Cc: HANSEN CHAN; Fred Baker; Naidu, Venkata; [email-masked]';
 > [email-masked]
 > Subject: RE: QoS in Shared Media
 > 
 > 
 > Jay Wang writes:
 > 
 >  > Well, RPR gives you bandwidth endurance at the link level, a shared
 >  > media in this case. On higher level, you may conduct your QoS as
 >  > appropriate, Diffserv, Intserv, or else. One does not preclude the other,
 >  > IMO.
 > 
 > how about if someone would like to give different bandwidth assurance to
 > more than one independent traffic class?  also, within a diffserv
 > traffic class, droping of packets is based on the drop presedence of the
 > packet, not based on behind which nodes the packets arrived.
 > 
 > as i said before, node based fairness can only be applied to the highest
 > dp packets within each class.  dpt (and also rpr if it adopts dpt) thus
 > supports diffserv only in a very limited sense.  as far as i remember,
 > it is the only ieee mac standard, where the specification tells how many
 > queues the switch has.
 > 
 > -- juha
 > 
 > 
 > -----Original Message-----
 > From: [email-masked] [mailto:[email-masked]]On Behalf Of Juha
 > Heinanen
 > Sent: Friday, May 04, 2001 12:01 PM
 > To: HANSEN CHAN
 > Cc: Fred Baker; Naidu, Venkata; [email-masked]'; [email-masked]'
 > Subject: Re: QoS in Shared Media
 > 
 > HANSEN CHAN writes:
 > 
 >  > How about the work in IEEE 802.17 RPR? I believe it will be a
 > shared-media
 >  > technology. Does that mean they will have a hard time to achieve QoS
 >  > guarantees?
 > 
 > last time that i checked, the rpr proposal (based on cisco dpt) had two
 > classes of service (strict priority and best effort). that is clearly
 > not adequate for diffserv.  node based fairness applied to the best
 > effort class.  in diffserv, node based fairness makes sense only for the
 > highest dp packets in each traffic class.
 > 
 > -- juha
 >

---

From: Fred Baker <[email-masked]>
Date: Mon, 30 Jul 2001 10:25:06 +0100

>node based fairness concept.

Juha: you and I have discussed this in private mail, and I at least 
partially disagree. Let me lay out the argument, and the assembled masses 
can comment.

It comes down to this: I will grant you that providing something as simple 
as two levels of priority does not provide all of the facilities of the 
diffserv architecture - you cannot control the rate of each data stream in 
the system if you are not given the knobs. This is not so very different 
from Ethernet, which even with 802.1q labelling doesn't give you rate 
controls on traffic streams within the switched part of the network.

However, apart from failure events, the SRP architecture provides a 
nominal-latency and loss-free domain, and allows for access control on 
injection of traffic into that domain. Thus, injection of traffic to the 
ring from any one system does not exceed the rates configured on those 
systems, and there is technology to manage babblers. I will argue that this 
is not significantly different from typical ISP backbones, which rely on 
bandwidth on their internal interfaces and provide QoS services only on 
their ingress/egress links if at all.

Are you telling me that in order to be diffserv compliant, not only must a 
network provide the technology at it's service interfaces, but also on each 
and every internal interface? If so, I don't predict early adoption...

Or are you telling me that to be compliant with the diffserv architecture, 
the device must specifically support the AF PHB on every interface? I don't 
find anything in the diffserv architectural documents that makes this 
stipulation. And even in the AF PHB (http://www.ietf.org/rfc/rfc2597.txt), 
while I see a stipulation that interfaces which might lose data are to drop 
randomly and prefer to drop from traffic marked "excess", I don't see any 
requirement that all interfaces drop.

I believe that you are essentially, in the final paragraph of your reply, 
asking that all switching be done at the IP layer, and that link layer 
networks of any sort be abandoned. I understand the viewpoint, but have to 
say that it is not completely practical; LAN technologies exist because 
they are inexpensive, useful, and widely used, and I don't know of *any* 
that support the diffserv architecture in the sense that you seem to want.

Following is the email I sent you on 5/10/2001, and your reply. For the 
remainder of the folks on the list, please excuse a few discussions of 
specific implementations which came up in this private conversation.

>To: Juha Heinanen <[email-masked]>
>From: Fred Baker <[email-masked]>
>Subject: [Fwd: QoS in Shared Media]
>Cc: Mike Takefman <[email-masked]>
>
>At 10:19 AM 5/9/2001, Juha Heinanen wrote:
>>the idea behind diffserv is that each traffic is allocated a certain
>>amount of aggregate resources (buffer space and bandwidth) at each
>>diffserv (l2 or l3) node.
>
>Mike Takefman, who manages the SRP hardware group, contacted me to be sure 
>he understood your issues and to see if we could sort them out. I think we 
>in fact have the support you need, but some of it is in the interface card 
>as opposed to being in the "Wide Area LAN" itslf. Let me walk through your 
>issues, the architecture they have proposed, and how I see this all 
>working, and then you can tell me where the disconnect is.
>
>If I think about ensuring a combination of voice/video a predictable 
>latency through EF and a set of AF classes for elastic traffic at various 
>rates, I wind up implementing a multi-level scheduler like so:
>
>         voice/video =====>||==================>priority scheduler
>                                             =>
>         AF class 1  =====>||====>\         /
>         AF class 2  =====>||====> \WRR/WFQ/
>         AF class 3  =====>||====> /Scheduler
>         AF class 4  =====>||====>/
>
>In short, I want to assure the voice/video traffic that it will not be 
>delayed by the elastic traffic, but will instead get predictable latency 
>at the interface, and I control rate using policing. For the elastic 
>traffic streams, I want to ensure each one a specific minimum rate, and 
>ensure that when arrival rate for a class exceeds departure rate, I signal 
>to a TCP or similar transport which can materially adjust its rate to 
>compensate using RED or WRED.
>
>What SRP provides is a collection of point to point duplex lines with 
>internal queuing which together comprise a ring. Each interface sees:
>
>                                            +-----+
>      high priority transit traffic in ->||-|     |
>      local high priority in ----------->||-|     |
>                                            |     |--->traffic out
>      low priority transit traffic in -->||-|     |
>      local low priority in ------------>||-|     |
>                                            +-----+
>
>Clearly, the other end of "traffic out" is "traffic in" at the next node, 
>which classifies traffic according to a three bit field copied from the IP 
>Precedence as high or low priority.
>
>The decision function favors high priority over low priority to ensure 
>minimal per-hop additional latency and jitter. The choice between local 
>and transit traffic is a bit more complicated. Each node on the ring is 
>guaranteed a certain lower bound bandwidth (at the moment, 1/N of the 
>ring, but Mike tells me that they are considering making this 
>specifiable), and if it is not using its fair share the decision function 
>will take local low priority traffic over low priority transit traffic, 
>and will take local high priority traffic over high priority transit 
>traffic. If the transit queues are too deep, however, it will take transit 
>traffic to clear the buffer. In the latter case, a longer term rule steps 
>in - the nodes communicate among themselves, and a system which fails to 
>get its share during one period of ten milliseconds will ask its neighbors 
>to back off for the next ten milliseconds. Hence, in any case, the low 
>priority has a predictable lower bound at each node.
>
>The systems which implement SRP also have extra chips which implement some 
>number of classes that use a WFQ/WRR type of scheduler, WRED, metering 
>with policing or marking, etc. The chip also allows a single queue to be 
>given absolute priority, which is appropriate for voice/video. These work 
>with any interface device, whether SRP, ATM, HSSI, POS, or whatever, which 
>is why the SRP guys don't think about them too much. Fundamentally, in 
>SRP's case, they divide the low priority capacity proportionally among a 
>set of traffic classes.
>
>I think you will agree that SRP with the external priority queue limits 
>edge to edge jitter and latency around the ring for the EF traffic. I want 
>to convince you that each AF class also gets a predictable lower bound 
>rate, and that RED drops apply when incoming traffic overruns available 
>capacity.
>
>I submit that if each node in the ring gets a predictable lower bound, and 
>subdivides that among its several AF classes proportionally, then each of 
>those sub-classes also has a predictable lower bound. This is for traffic 
>entering the ring from each node; how it leaves the ring depends on the 
>traffic itself, and in the worst case stays on the ring all the way 
>around. Thus, the proportions on the ring itself may vary, but that is not 
>necessarily an SLA objective - the SLA objective is to give each edge 
>device its due, and that is done.
>
>In discussing this with the SRP team, I proposed a scenario in which I 
>wondered if traffic from one node could be reduced or locked out by the 
>preponderance of traffic from other nodes. That's where the "I'm not 
>getting my share" message comes in; even in that case, the other nodes 
>slowing down should result in the restricted node getting its bandwidth 
>needs met, by forcing competing traffic to back up into the WRR/WFQ queues 
>on the competing nodes. Any place that traffic backs up into a WRR/WFQ 
>queue, RED/WRED can be applied to talk with those TCP senders, so RED 
>happens and it happens to the right set of senders.
>
>It seems to me that we have all the bases covered. They are not covered 
>within the SRP component on the interface card, but they are covered in 
>the queuing components that sit in front of the SRP interface on the line 
>card. In other words, the following happens on the line card:
>
>         voice/video =====>||==================>priority scheduler
>                                             =>
>         AF class 1  =====>||====>\         /
>         AF class 2  =====>||====> \WRR/WFQ/
>         AF class 3  =====>||====> /Scheduler
>         AF class 4  =====>||====>/
>
>and on the SRP ring, a simple priority scheduler preserves the 
>latency/jitter characteristics needed by EF, and each elastic traffic 
>class (from each node and from the aggregate of the nodes) is guaranteed a 
>predictable lower rate bound.
>
>Am I missing something?

>From: Juha Heinanen <[email-masked]>
>To: Fred Baker <[email-masked]>
>Cc: Mike Takefman <[email-masked]>, [email-masked]
>Subject: [Fwd: QoS in Shared Media]
>Date: Thu, 10 May 2001 15:46:30 +0300
>
>fred,
>
>thanks for your thorough explanation.  my comments follow.
>
>queuing at two levels (layer 3 and 2) is always problematic.  we have
>learned this from atm, where l3 diffserv queueing and scheduling is
>messed up when the properly scheduled packets enter the atm layer's per
>vc queues. i'm afraid that a similar problems exist with rpr if rpr
>doesn't do same kind of queueing/scheduling using the user priority
>field as is done at l3 using the precedence field.
>
>based on your description i understand that in rpr the low priority
>capacity is allocated fairly to each node, i.e., each node has a lower
>bound on the amount of low priority traffic and can then distribute it
>to the elastic traffic classes according to its policy.  this doesn't,
>however, mean that a given traffic class would IN THE RING ITSELF get a
>given proportion of bandwidth, because one node could, for example, send
>only best effort packets and they would get as much ring bandwidth as
>business class packets from another node.
>
>in other words, it seems to me that diffserv and node based fairness and
>separation of local and transit traffic simply do not match. if we
>forget node based fairness and local/transit traffic it could be
>possible to end up with a proper diffserv implementation where l2
>packets entering the ring (first time or transit) would be all
>queued/scheduled/dropped by the rpr interface based solely on the
>traffic class and drop precedence coded in the user priority field.
>additional complications may result from the ring wraps, but i don't
>remember the details.
>
>-- juha

---

From: Juha Heinanen <[email-masked]>
Date: Mon, 30 Jul 2001 20:33:08 +0300 (EEST)

fred,

i have no problems whatsoever with diffserv support in regular 801.1q
ethernet switches that allow me to map the user priority field to wfq
queues.  rpr doesn't offer this capability. 

phb stands for "per HOP behavior".  of course it is possible to support
diffserv only at the edge, but that doesn't lead to the most efficient
use of the bandwidth.  it is the same as using cbr vc to connect routers
together over atm backbone.

-- juha

---

From: Christian Kuhtz <[email-masked]>
Date: Mon, 30 Jul 2001 13:55:47 -0400

> together over atm backbone.

Err, lets set the perfect world aside for a moment.  

If your core has a specific capacity, and you meter, classify, policy at the
edge, don't you have guarantees which enable differentiated services?  Sure,
diffserv everywhere would be cool.. but, also $$$ and how real is this real?

You can drive diffserv as far as you want, and theoretically anything's
possible, but at some point you reach the point of diminishing returns.. 

Budgetary realities do exist.

Flame away.

Cheers,
Chris

-- 
Christian Kuhtz <[email-masked]> -wk, <[email-masked]> -hm
Sr. Architect, Engineering & Architecture, BellSouth.net, Atlanta, GA, U.S.
"I speak for myself only.""

---

From: "Joel M. Halpern" <[email-masked]>
Date: Mon, 30 Jul 2001 14:02:45 -0400

[Given that I did not start this, I am reluctant to change the target wg, 
but it sure looks like a diffserv discussion, not an MPLS or TE discussion.]

Juha, given that your earlier note said specifically that the technology 
under discussion does support two levels of priority queueing,  it seems 
that you do not have any grounds to complain that it does not support 
diffserv.  It does not support the particular PHB the way you want it 
supported.  One could have a discussion about whether it supports a given 
PHB at all under a specific configuration or under all possible 
configurations.  But that does not mean that the technology does not 
support or is not compliant with diffserv.

Yours,
Joel M. Halpern

---

From: Eric Gray <[email-masked]>
Date: Mon, 30 Jul 2001 14:14:11 -0400

Chris,

    Unless the drain rate at each of your egress points exceeds the
capacity of your core, you don't achieve any particularly good
guarantees if all you do is police at the edge.  In fact, what you
might see is worst case behavior where a lot of traffic makes it close
to an egress point and is dropped as a result of local congestion AFTER
having already consumed core resources in getting that far.

--
Eric Gray

---

From: Juha Heinanen <[email-masked]>
Date: Mon, 30 Jul 2001 21:13:56 +0300 (EEST)

> Budgetary realities do exist.

i don't get this point.  any cisco or juniper core router supports
diffserv equally well or badly over pos, serial and ethernet interfaces.
also all decent ethernet switched mapping of user priority to wfq
queues.  this is not thus not a money issue, but rpr issue.

-- juha

---

From: Juha Heinanen <[email-masked]>
Date: Mon, 30 Jul 2001 21:17:52 +0300 (EEST)

joel,

there is no such thing as "diffserv compliance".  to me a device is
diffserv capable if i can map a layer 3 or layer 2 "tos" code point to
a wfq and red drop precedence.  as my example showed, dpt/rpr doesn't
have this capability.

-- juha

---

From: Fred Baker <[email-masked]>
Date: Mon, 30 Jul 2001 10:50:23 +0100

At 06:33 PM 7/30/2001, Juha Heinanen wrote:
>phb stands for "per HOP behavior".

yes, and it is supported at every IP hop.

---

From: Fred Baker <[email-masked]>
Date: Mon, 30 Jul 2001 11:36:44 +0100

At 07:14 PM 7/30/2001, Eric Gray wrote:
>guarantees if all you do is police at the edge.

the drain rate of an RPR ring is presumably the same is its internal rate - 
if all the traffic is directed to a particular system, it will drain all 
the traffic from the ring. What it does on the next interface is 
interesting from a larger perspective, but not from the perspective of the 
RPR ring.

The point here is that diffserv may be used to control the rates at which 
various RPR systems put various kinds of traffic onto the RPR ring, and RPR 
will carry it. Like diffserv in any network, if supply exceeds demand 
sometime later, we can only hope that the users' TCPs will notice that and 
slow down.

---

From: Fred Baker <[email-masked]>
Date: Mon, 30 Jul 2001 11:41:30 +0100

At 07:17 PM 7/30/2001, Juha Heinanen wrote:
>to me a device is diffserv capable if i can map a layer 3 or layer 2 "tos" 
>code point to
>a wfq and red drop precedence.  as my example showed, dpt/rpr doesn't have 
>this capability.

There is no requirement that a diffserv device implement WFQ. Hence, this 
is not fundamental.

In a system which drops no traffic, what does "drop preference" mean?

Yes, RPR does not internally implement WFQ or RED. It does offer the 
ability to differentiate among streams of traffic based on an attribute 
carried in the packet without having to perform obnoxious ACLs. Those are 
the only requirements I find in RFCs 2474/2475.

---

From: Juha Heinanen <[email-masked]>
Date: Mon, 30 Jul 2001 21:56:29 +0300 (EEST)

> There is no requirement that a diffserv device implement WFQ. Hence, this 

it is pretty hard to implement af without wfq and red or something that
closely resembles them.

 > In a system which drops no traffic, what does "drop preference" mean?

nothing, but bandwidth (especially international) so far has not been
free.

i can't quite figure out why this outburst to my email, where i gave a
simple example on how rpr doesn't support af.  i very well understand
that not everyone needs that capability and is happy with rpr rings.

-- juha