Subject: [Capwap] capwap transport analysis: QoS vs multiport

From: "Scott G. Kelly" <[email-masked]>
Date: Fri, 9 Jun 2006 15:13:48 -0700 (GMT-07:00)

We've been debating the transport facility for capwap for months now, and have yet to reach closure. We, as a working group, need to choose one of these approaches and move on. My preference is that we choose the one that makes the most sense technically, but many years of IETF experience suggest that this is not always the way these things work out. Still, I think we should try to do the right thing here since we basically have a clean slate right now, and we will never have this opportunity again. To that end, I will once more attempt to review the salient arguments on either side.

If I am missing something, please bonk me over the head and show me what it is, and I will gladly modify my position accordingly - I really do want to choose the approach that best serves the broadest range of customers. I am not irrevocably committed to the single port approach, but detailed analysis has persuaded me that it's the best approach, and that the two-port approach simply does not accomplish what is claimed. If you can specifically show that the analysis is flawed, I will accept this and adapt my thinking accordingly.

I'll share some more of that analysis here, but I won't replicate the previous text - you can flip back through the archives for that. Here I will talk about QoS only. This will be part of the record of our considerations and decision making process, so please read it, think carefully, and comment constructively.

On behalf of the two-port approach, the primary argument pertains to traffic engineering, the starting premises being

(1) intermediate network elements between the WTP and AC must be able to re-mark packets

(2) in particular, they need to ensure that control traffic is treated at a higher priority than data traffic

(3) some customers will want to send control down a different path than data to achieve this (or some other) objective

(4) many of these elements can only classify traffic based on 5-tuples; they apparently cannot use VLAN tags or 1:1 mappings of DSCP/802.1q/802.1d mappings

Based on these premises, the claim is that using separate ports for control and data will provide a solution. Let's assume that the various premises and claim are correct, and see where it leads us. We can start by taking a closer look at the traffic in question:

Control Channel Content
----------------------------
- Session establishment messages (discover, dtls handshake, join)

- configuration changes (includes config updates, radio up/down)

- echo request/replies

- periodic stats/status updates

If we expect the network to be stable (the average or common case if we've done our jobs), then capwap session establishment is a miniscule fraction of total control traffic, as are firmware updates and other config changes. Once the capwap session is established and the WTP is configured (steady state), control channel traffic will be relatively sparse, compared to data traffic. Echo requests/replies probably wouldn't be more frequent than one per second per WTP (and in many cases will be much less frequent than this), and stats updates should be much less frequent than echoes (but it depends on the implementation).

The cost of control packet loss depends upon contiguous sequential echo req/rsp losses exceeding the NeighborDeadInterval timer. If you lose one or both of the echo req/rsp packets in every exchange over one of these entire intervals, the capwap "connection" will be considered lost, and will be reset. Note the tuning ability this provides.

Also note that if this is happening, your wlan-to-wired applications are faring quite poorly regardless of whether this connection is reset, although a reset is definitely to be avoided if possible - but unless this is an anomaly, you have a longer-term problem to solve anyway, because users will not be happy with a jerky, bumpy, unreliable network like the one we're describing.

With respect to control channel QoS, the most important thing is to prevent the channel from being incorrectly diagnosed as "dead" when in fact it is not. With two ports, you could conceivably give the control channel a higher priority so that this happens less frequently on a highly-congested network, but then what happens to all of the QoS granularity in the data channel? 

Data Channel Content
-------------------------
- 802.11 management frames
  - probes (usually not sent to AC, so let's ignore for now)
  - association
  - authentication

- client data
  - voice
  - file transfers
  - netbios chatter
  - other broadcast traffic
  - video streaming
  - web surfing
  - imap/email
  - instant messaging
  - (etc)

Since you only have a single port (and the claim is that it's too hard to look deeper into the packets), then you will be effectively "clipping" the QoS level of any data streams requiring a higher priority than you are re-marking with, and "promoting" any data with a lower priority. 

Notice that some of the 802.11 messages (in particular, association messages) are quite timing critical. Also, what about *voice* packets?

Am I the only one who thinks this is broken? One of our objectives (resource control objective) is to coordinate quality across wireless and wired segments. I won't quote chapter and verse (like I said, it's not scripture or anything), but I think we all agree that this is an important objective. We are throwing that to the wind here, and this will definitely have an adverse impact on the applications in the data channel (or, put slightly differently, on the quality of the products we provide).

In order to meet the resource control objective, and perhaps more importantly, provide customers with WLANs that really work for timing-sensitive applications, we have to support granular QoS between the WTPs and ACs which more or less matches the over-air QoS. Using a single port as one-size-fits-all QoS classifier for the data channel is diametrically opposed to this objective. 

There is no question that there are QoS problems we have yet to solve, but using a port for control and a port for data simply does not do what we are asking of it. So, what else can we do? One option is to provide a separate data channel port per QoS class, but given the channel binding issues, nat traversal issues, firewall issues, and associated implementation complexity, this is going from bad to worse.

Over the air, packets may be marked (e.g. via WMM, 802.11e), and in order to match these service levels on the wired segment, the AC or WTP can map these markings to the outer header of the individual data channel packets, either as 802.1q/d, or DSCP values, but this (like any other scheme we come up with) requires agreement between the WTP/AC and the owner(s) of the media between them. And of course, you have to "trust" the applications not to lie, and self-promote. 

As for the first issue (agreement), this is facilitated by providing the appropriate knobs on the AC/WTP. As for the second issue (self-promotion), while it sounds very negative, it is not really very likely, and maybe not even worth the cost of prevention. However, if applications were innocently using the "wrong" markings because they didn't know any better, this could be remedied by providing granular classification at the WTP or AC, or maybe even per-client re-marking (or perhaps via buying applications that work "right"). Still, it doesn't seem worth it to spend many cycles on what should be a rare problem. A third alternative is to use granular classification in the cloud between the AC and WTP, but clearly, this requires deep packet inspection (which we accepted as not being available in our premises above).

No matter what, the original claim (that two capwap ports will resolve the QoS issues) is clearly wrong. There are other solutions, but none of them are perfect. Multiple ports (one per QoS class) is a possibility, but I shudder to think of it. AC/WTP marking seems like the obvious solution and the only other alternative I can see is in-the-cloud deep packet inspection. Either of the more viable solutions favors the single-port approach, given the complexity reduction, built-in nat traversal, firewall rule reduction it entails, and all other things being equal.

--Scott

_________________________________________________________________
To unsubscribe or modify your subscription options, please visit:
http://lists.frascone.com/mailman/listinfo/capwap

Archives: http://lists.frascone.com/pipermail/capwap

---

From: "Bob O'Hara (boohara)" <[email-masked]>
Date: Fri, 9 Jun 2006 16:48:03 -0700

Scott,

*BONK* Yes, you are missing everything.  Your argument appears to be "if I can't do perfect QoS, it is not worth trying to do any QoS at all."  Perhaps you hadn't yet realized that this is where your arguments were taking you.  I hope this email helps to clarify that.  

This line of argument, of course, leads one to conclude that a single port is all that is needed for CAPWAP, since perfect QoS on the data and control packets of CAPWAP can't be done anyway.  The only apparent solution to this is to use as many ports as there are different levels of QoS needed for the CAPWAP control and for the 802.11 data tunneled in CAPWAP data packets, as Scott described below.  That, of course, is absurd and I agree with Scott.

The line of argument presented below is an attempt at reductio ad absurdum.  Given some set of absurd conditions, the method of using multiple ports falls apart.  You use this to justify your conclusion that two ports are not useful.  

Simply because we can't do something perfectly, is no reason not to do the best that we can.  Certainly we can agree that some QoS is better than none, which is what the two port solution gets us.

Also, please see my specific comments, below.  

 -Bob
 
-----Original Message-----
From: Scott G. Kelly [mailto:[email-masked]] 
Sent: Friday, June 09, 2006 3:14 PM
To: capwap
Subject: [Capwap] capwap transport analysis: QoS vs multiport

[snipped to prevent the killing of more bits than absolutely necessary]

Bob>> The statement has been made by several posters not that the equipment cannot do this, but that it will not do it, because the source of any QoS marking at the WTP may not be trusted.  The reason for this is that the source of this QoS marking is the client, not the WTP and the client is not trusted to mark QoS properly.  Therefore any mapping of that QoS by the WTP cannot be trusted, either.  More on this, below.

Control Channel Content

- echo request/replies

Bob>> Correct, there is tuning that can be done here.  This is apparently an allusion to the fact that one can make this timer value as large as one wants to be able to live with as many lost echo packets as one can conceive.  This is true.  The cost of that is that one also has to live with that same length of time before a truly lost connection is recognized and the WTP can make an attempt to find another AC.  So, the cost of losing the ability to reduce the probability of losing these packets by differential QoS is a longer network outage for all users when the connection to the AC is truly gone.

Bob>> Apparently this argument is that no network connection for the client devices is better than one that is not perfect.  I would take issue with that.  Getting some data through the network may not be able to support all applications.  But, it will support some.  This is certainly better than having a device with no network connection at al.

Data Channel Content

- client data
  - voice
  - file transfers
  - netbios chatter

Bob>> This is no different from the QoS provided to the data packets when using the mux header.  Given that the frame for this discussion is that the equipment between the WTP and AC does not trust the WTP marking of the QoS, how could it be any different?

Bob>>What is claimed here is that this two-port method is broken, when the single-port method that provides even less differentiation for CAPWAP packets is claimed to not be broken.  It would be quite interesting to understand how this could be so, again given that the same constraints are applied to both.

Bob>>I see you have finally argued yourself into the corner.  Let me quote you: "using a single port as one-size-fits-all QoS classifier" is exactly what your preferred solution does to an even greater degree than the two-port solution.

Bob>>Exactly the point.  Trusting the applications is precisely the problem.  In a perfect world, we would have no worries about an application self-promoting its own QoS at the expense of other applications or other users of the network.  Unfortunately, that perfect world is a pipe dream.  The real world, the world we need to design our protocol for must live with the fact that the applications' markings for QoS will not be trusted by the network.  Period.  Anything else on our part is extremely naïve.

As for the first issue (agreement), this is facilitated by providing the appropriate knobs on the AC/WTP. As for the second issue (self-promotion), while it sounds very negative, it is not really very likely, and maybe not even worth the cost of prevention. 

Bob>> This is much too naïve.  Next thing you will be advocating is that we don't need strong authentication or encryption, because the users and their machines can be trusted not to lie about who they are.  Why is this trust issue any different?

However, if applications were innocently using the "wrong" markings because they didn't know any better, this could be remedied by providing granular classification at the WTP or AC, or maybe even per-client re-marking (or perhaps via buying applications that work "right"). Still, it doesn't seem worth it to spend many cycles on what should be a rare problem. A third alternative is to use granular classification in the cloud between the AC and WTP, but clearly, this requires deep packet inspection (which we accepted as not being available in our premises above).

Bob>> Well, this might be an option, if one could see the data in the 802.11 frame.  Unfortunately, a CAPWAP option allows all of that to be obscured from all of the infrastructure, except for the AC.  Utilizing centralized encryption negates either of the two alternatives described above.  Neither the WTP nor the equipment between the WTP and the AC can classify the packet based on the content of the payload (the now-encrypted 802.11 frame).

Bob>> As has been shown by the arguments from the original email arguments above, everything claimed that is better than using separate ports for control and for data is actually worse, including using a single port and a mux header.

--Scott

---

From: "Scott G. Kelly" <[email-masked]>
Date: Fri, 9 Jun 2006 17:36:30 -0700 (GMT-07:00)

Hi Bob,

>*BONK* Yes, you are missing everything.  Your argument appears to be "if I can't do perfect QoS, it is not 
> worth trying to do any QoS at all."  Perhaps you hadn't yet realized that this is where your arguments were
> taking you.  I hope this email helps to clarify that.  

No, the argument is nothing of the kind - it is simply that the multiport approach does nothing for QoS that WTP/AC marking cannot. If you trust the ports (which the WTP and AC fill in), why can't you trust the QoS markings that you've configured them to provide? Set aside the client marking concerns you have (assume we won't trust clients to mark), and instead assume the WTP uses two markers, one for control and one for data. How is this different or worse than separate ports from a QoS perspective? It is clearly far better than two ports from an operational and implemenational complexity perspective.

>This line of argument, of course, leads one to conclude that a single port is all that is needed for CAPWAP, since perfect QoS on the data and control packets of CAPWAP can't be done anyway.  The on…

It's not that it can't be done - it's that you have the same problem with one or two ports, and *that* is the problem we should be expending energy on, not justifying an approach whose intended function really has nothing to do with QoS. I'm sorry it's not the answer you wanted, but the facts seem pretty clear, and I don't think you're effectively refuting them.

>The line of argument presented below is an attempt at reductio ad absurdum.  Given some set of absurd conditions, the method of using multiple ports falls apart.  You use this to justify your conclu…

Actually, I agree that it does reduce the two-port approach _as a proposed QoS solution_ to be... well, I wouldn't have used the term absurd - but I might say detrimental.

>Simply because we can't do something perfectly, is no reason not to do the best that we can.  
> Certainly we can agree that some QoS is better than none, which is what the two port solution
> gets us.

I agree that some is better than none, and I think it's clear that we can do better.  I think that is explained at the end of the post. I tried to avoid writing a book about it, but I think there's enough there.

>Also, please see my specific comments, below.  

Yes, I read them, and I think it would be less than productive to do a blow-by-blow reply. You clearly feel quite strongly about this. Let's let everyone else review the post on their own, and see what they think. Chill out, have a nice weekend, and we can discuss on Monday.

Scott

---

From: "Bob O'Hara (boohara)" <[email-masked]>
Date: Mon, 12 Jun 2006 08:42:11 -0700

Scott,

Of course you can't respond to my specific arguments, because they show
your line of reasoning to be without any merit.  Let's just leave it at
that.

 -Bob
 
-----Original Message-----
From: Scott G. Kelly [mailto:[email-masked]] 
Sent: Friday, June 09, 2006 5:37 PM
To: Bob O'Hara (boohara); capwap
Subject: Re: [Capwap] capwap transport analysis: QoS vs multiport

>Also, please see my specific comments, below.  

Yes, I read them, and I think it would be less than productive to do a
blow-by-blow reply. You clearly feel quite strongly about this. Let's
let everyone else review the post on their own, and see what they think.
Chill out, have a nice weekend, and we can discuss on Monday.

Scott

---

From: "Scott G. Kelly" <[email-masked]>
Date: Mon, 12 Jun 2006 09:50:49 -0700 (GMT-07:00)

Hi Bob,

>your line of reasoning to be without any merit.  Let's just leave it at

My original post was quite long and detailed, and I thought it might be a bit tedious to subject everyone to a blow by blow response immediately after posting that. This is a complex discussion, and it's important to keep people engaged throughout the reasoning process. We've had the weekend to think this over, and hopefully other folks have had a chance to review both the post and your response and form opinions of their own by now. So, let's resume the discussion.

I think the most effective way to deal with this is systematically. The starting premises are the foundation of what follows, so it they are incorrect, the issue must be re-analyzed in light of this. If we can approach this in a disciplined manner, we should be able to get through this quickly.

Let's start with the premises and the comment immediately following them:

> (2) in particular, they need to ensure that control traffic 
> is treated at a higher priority than data traffic
> 
> (3) some customers will want to send control down a different 
> path than data to achieve this (or some other) objective
> 
> (4) many of these elements can only classify traffic based on 
> 5-tuples; they apparently cannot use VLAN tags or 1:1 
> mappings of DSCP/802.1q/802.1d mappings
> 
> Bob>> The statement has been made by several posters not that 
> the equipment cannot do this, but that it will not do it, 
> because the source of any QoS marking at the WTP may not be 
> trusted.  The reason for this is that the source of this QoS 
> marking is the client, not the WTP and the client is not 
> trusted to mark QoS properly.  Therefore any mapping of that 
> QoS by the WTP cannot be trusted, either.  More on this, below.

I assume this is meant to refute premise (4), based on the supposition that the only way to mark traffic is by copying WMM/.11e markings. This is not the case, as we know that WTPs and ACs can be programmed to classify traffic and mark it accordingly. 

Scott

---

From: "Bob O'Hara (boohara)" <[email-masked]>
Date: Mon, 12 Jun 2006 16:40:23 -0700

Scott,

[snip]

Let's start with the premises and the comment immediately following

> QoS by the WTP cannot be trusted, either.  More on this, below.

I assume this is meant to refute premise (4), based on the supposition
that the only way to mark traffic is by copying WMM/.11e markings. This
is not the case, as we know that WTPs and ACs can be programmed to
classify traffic and mark it accordingly. 

Bob>> This statement is to refute the claim made in premise (4) that the
equipment cannot use VLAN tags or mappings of the DSCP/802.1Q/802.1D
information in the frame/packet received from the 802.11 client device.
The equipment could use it, if the information were trusted.  The
information is not trusted.  Therefore, the information cannot be used.

 -Bob

---

From: "Scott G. Kelly" <[email-masked]>
Date: Mon, 12 Jun 2006 17:05:15 -0700 (GMT-07:00)

Hi Bob,

>information is not trusted.  Therefore, the information cannot be used.

[I don't agree that we should universally mistrust client marking, but that is a different (and independent) discussion, so we can take that up later.]

So if the AC and WTP were to mark control packets so as to make them distinguishable from data packets (using one of the marking methods suggested above, and without relying on client truthfulness), what problems remain? 

Scott

(4)

---

From: "Bob O'Hara (boohara)" <[email-masked]>
Date: Tue, 13 Jun 2006 08:11:31 -0700

Scott wrote:

>So if the AC and WTP were to mark control packets 
>so as to make them distinguishable from data packets 
>(using one of the marking methods suggested above, 
>and without relying on client truthfulness), what 
>problems remain? 

As Pat points out in a separate email, the problem of controlling how
the WTP marks those packets remains to be solved.  But, that discussion
can continue in Pat's thread.

Another problem is that the WTP, itself, is not trusted by the network
to which it is attached.  A widespread example is a network of WLAN
hotspots.  The WTPs at the hotspot are connected to the AC over a third
party's network.  

In some of those hotspots, the WTP will be local-MAC.  A hotspot user's
packets from a local-MAC WTP will be able to be inspected at ingress to
the third party's network, since the 5-tuple is clearly available in
those packets.  QoS can be applied to those user data packets, without
requiring any changes to the third party's network equipment.

However, if those same packets are sent to the AC by a split-MAC WTP,
the third party network will not be able to distinguish control packets
from data packets, unless they are sent on separate ports.  In the
CAPWAP packet, the 5-tuple is the same for both control and data.

 -Bob

---

From: "Bob O'Hara (boohara)" <[email-masked]>
Date: Wed, 14 Jun 2006 10:42:27 -0700

Scott, 

Your reformulation of premise (4) is closer to my intent, but still off
the mark.  This is what I would prefer to see from premise (4):

(4) many of these elements only classify traffic based on 5-tuples, due
to administrative or policy constraints; they will not use
AC/WTP-provided VLAN tags or DSCP/802.1q/802.1d markings for QoS
purposes

 -Bob
 
-----Original Message-----
From: Scott G. Kelly [mailto:[email-masked]] 
Sent: Tuesday, June 13, 2006 10:42 AM
To: Bob O'Hara (boohara); capwap
Subject: Re: [Capwap] capwap transport analysis: QoS vs multiport

Hi Bob,

>CAPWAP packet, the 5-tuple is the same for both control and data.

Keeping in mind that what we are trying to do is ascertain whether
premise (4) was correctly formulated, let me re-state that in its
original form:

(4) many of these elements can only classify traffic based on 5-tuples;
they apparently cannot use VLAN tags or 1:1 mappings of
DSCP/802.1q/802.1d mappings

Apparently, what is at issue is the phrase "1:1 mappings of" - if I
apply what you've said above to this premise, it seems that this would
be an acceptable re-formulation:

(4) many of these elements can only classify traffic based on 5-tuples;
they apparently cannot use AC/WTP-provided VLAN tags or
DSCP/802.1q/802.1d markings for QoS purposes

Does this reflect your intent?

Thanks,

Scott