Subject: [Capwap] reasonably bounding our QoS ambitions for the base protocol

From: "Scott G. Kelly" <[email-masked]>
Date: Tue, 13 Jun 2006 20:27:24 -0400 (EDT)

I wanted to limit my earlier reply to the question the thread originally was addressing (getting the premises straight), but I also think the response below needs more attention. I've changed the subject line to keep the two threads distinct.

Comments inline...

-----Original Message-----
>From: "Bob O'Hara (boohara)" <[email-masked]>
>Sent: Jun 13, 2006 11:11 AM
>To: capwap <[email-masked]>
>Subject: Re: [Capwap] capwap transport analysis: QoS vs multiport
>
>
>
>Scott wrote:
>
>>So if the AC and WTP were to mark control packets 
>>so as to make them distinguishable from data packets 
>>(using one of the marking methods suggested above, 
>>and without relying on client truthfulness), what 
>>problems remain? 
>
>As Pat points out in a separate email, the problem of controlling how
>the WTP marks those packets remains to be solved.  But, that discussion
>can continue in Pat's thread.
>
>Another problem is that the WTP, itself, is not trusted by the network
>to which it is attached.  A widespread example is a network of WLAN
>hotspots.  The WTPs at the hotspot are connected to the AC over a third
>party's network.  
>
>In some of those hotspots, the WTP will be local-MAC.  A hotspot user's
>packets from a local-MAC WTP will be able to be inspected at ingress to
>the third party's network, since the 5-tuple is clearly available in
>those packets.  QoS can be applied to those user data packets, without
>requiring any changes to the third party's network equipment.

Currently, the vast majority (if not all) of the deployed hotspots are local MAC, and users of those hotspots have no expectation of QoS. If the access point requires QoS for control traffic, then it also needs it for the 802.11 mgmt traffic that it's forwarding to the AC, and the hotspot provider would negotiate an appropriate agreement with the ISP - but in this case, *both* the capwap control and data channels would require QoS. However, there is a much simpler solution (assuming this ever becomes a real problem).

If user data *were* choking the Internet link and hence causing control traffic starvation, the WTP is in a unique position to detect this and defend itself by throttling down user traffic while it attempts to re-establish its control linkage. Also, there is no requirement that a local-MAC WTP reset just because the control channel is temporarily lost - indeed, isn't it common (in fact, demanded by customers) today for local MAC WTP's to switch into a restricted mode upon loss of the control channel connection, and to maintain existing user sessions while rejecting new ones? That is, this particular problem is already being effectively addressed in other ways.
 
>However, if those same packets are sent to the AC by a split-MAC WTP,
>the third party network will not be able to distinguish control packets
>from data packets, unless they are sent on separate ports.  In the
>CAPWAP packet, the 5-tuple is the same for both control and data.

I'm having a really hard time with this one - forwarding user data back to the AC in a hotspot scenario, and doing it over an untrusting (and untrusted) 3rd party network, and expecting to provide QoS to boot? It's hard to imagine this happening in practice, but the fact is that the same argument applies with respect to control channel traffic, i.e. if a flooded data channel is causing control channel starvation, the WTP or AC *knows* this - it's immediately evident - and the WTP or AC can throttle back the data channel flow in favor of the control channel. And this requires no 3rd party intervention.

I'm sure that if we put our minds to it, we can dream up corner cases which might not be addressed by a single port protocol,  but there are other ways to solve the problems above that don't entail the complexity and costs of the multiport approach. And as an aside, this would appear to be one of the cases where NAT is most likely to occur, further complicating things. 

----

I think it's important to acknowledge that none of the proposed solutions is perfect, or will service all possible scenarios that we might dream up. I think it's also important to realize that there are many problems to be solved in wlan deployments, and we are not, indeed cannot be chartered to solve each and every one of them here. We need to solve a reasonable subset of them, and address other problems in later efforts. 

The vast majority of real-world deployments where reliable provision of QoS might be reasonably expected are within a single domain of control, and those that are not present very challenging problems - problems which should not within the charter of this working group at this time, if we are to accomplish anything useful within a meaningful time frame. 

Scott

_________________________________________________________________
To unsubscribe or modify your subscription options, please visit:
http://lists.frascone.com/mailman/listinfo/capwap

Archives: http://lists.frascone.com/pipermail/capwap

---

From: "Pat Calhoun (pacalhou)" <[email-masked]>
Date: Tue, 13 Jun 2006 17:48:18 -0700

> The vast majority of real-world deployments where reliable 
> provision of QoS might be reasonably expected are within a 
> single domain of control, and those that are not present very 
> challenging problems - problems which should not within the 
> charter of this working group at this time, if we are to 
> accomplish anything useful within a meaningful time frame. 

Scott,

Either way you slice this issue, there is work on the table. If we
opt for the MUX, then we need to add packet classification extensions
to the protocol. If we opt for multi-port, then we add a keepalive to
the data channel and text to ensure the state machine is fine. 

Personally, I think multi-ports opens up significant scenarios that
are not otherwise available in the MUX case. Given we have work either
way, I don't see why the MUX proposal is more attractive. All it does
Is add restrictions, which will cause the WG to solve this problem in
a subsequent revision of the RFCed protocol (-bis, or ver 2), in a
manner that will certainly create header imcompatibilities.

Why walk into this today, when we know pain will come from it? 

Nothing is free.

Pat Calhoun
CTO, Wireless Networking Business Unit
Cisco Systems
_________________________________________________________________
To unsubscribe or modify your subscription options, please visit:
http://lists.frascone.com/mailman/listinfo/capwap

---

From: "Scott G. Kelly" <[email-masked]>
Date: Tue, 13 Jun 2006 21:10:37 -0400 (EDT)

Hi Pat,

>the data channel and text to ensure the state machine is fine. 

I think this misses the mark by quite a bit. No matter what we do, we have QoS-related packet classification issues. That should be crystal clear by now.  QoS is not just a question of distinguishing capwap control from data; we need granular QoS for encapsulated data channel elements, and perhaps even more so. Neither solution provides this, and that is the *real* QoS work that needs doing. The multiport proposal does nothing to address these issues, but will cost us a lot for an arguably non-existent benefit. Honestly, after looking closely at this, I find it hard to believe that QoS is the real motivator here.

>manner that will certainly create header imcompatibilities.

It is clear from analysis that has been posted here that the multiport approach entails far more pain than the single port approach, and solves far fewer problems in the process. Neither approach is perfect, but the multiport approach will clearly have numerous and significant operational issues that the single port approach will not. These issues have been documented and remain unrefuted. Ignoring facts does not make them go away.

>Why walk into this today, when we know pain will come from it? 

Clearly, and we should choose the approach that maximizes the return on our investment. Splitting capwap into two sub-protocols to partially (and poorly) solve problems that have already have lower cost solutions seems like a bad bet.

Scott

---

From: "Pat Calhoun (pacalhou)" <[email-masked]>
Date: Wed, 14 Jun 2006 08:55:55 -0700

Scott, we've entered a circular logic world. I've addressed each one of 
your points in various e-mails, not ignored them and hope they go away,
as you state.

I, and others, have raised QoS *and* scalability issues with the
proposal,
and the points raised on scaling haven't been refuted - or even
answered.
I've stated that this proposal will require us to go down the multi port
option in the future anyhow, and all we are doing is avoiding up front 
pain for long term interoperability issues.

Maybe it's just me, but it appears you are simply not willing to listen
to arguments against your MUX proposal. I don't think there's much that
can be said at this point, other than to agree that we disagree.

Pat Calhoun

> From: Scott G. Kelly [mailto:[email-masked]] 
> Sent: Tuesday, June 13, 2006 6:11 PM
> To: Pat Calhoun (pacalhou); Bob O'Hara (boohara); capwap
> Subject: RE: [Capwap] reasonably bounding our QoS ambitions 
> for the base protocol
> 
> Hi Pat,
> 
> >Scott,
> >
> >Either way you slice this issue, there is work on the table. 
> If we opt 
> >for the MUX, then we need to add packet classification extensions to 
> >the protocol. If we opt for multi-port, then we add a 
> keepalive to the 
> >data channel and text to ensure the state machine is fine.
> 
> I think this misses the mark by quite a bit. No matter what 
> we do, we have QoS-related packet classification issues. That 
> should be crystal clear by now.  QoS is not just a question 
> of distinguishing capwap control from data; we need granular 
> QoS for encapsulated data channel elements, and perhaps even 
> more so. Neither solution provides this, and that is the 
> *real* QoS work that needs doing. The multiport proposal does 
> nothing to address these issues, but will cost us a lot for 
> an arguably non-existent benefit. Honestly, after looking 
> closely at this, I find it hard to believe that QoS is the 
> real motivator here.
> 
> >Personally, I think multi-ports opens up significant 
> scenarios that are 
> >not otherwise available in the MUX case.  Given we have work either 
> >Is add restrictions, which will cause the WG to solve this 
> problem in a 
> >subsequent revision of the RFCed protocol (-bis, or ver 2), 
> in a manner 
> >that will certainly create header imcompatibilities.
> 
> It is clear from analysis that has been posted here that the 
> multiport approach entails far more pain than the single port 
> approach, and solves far fewer problems in the process. 
> Neither approach is perfect, but the multiport approach will 
> clearly have numerous and significant operational issues that 
> the single port approach will not. These issues have been 
> documented and remain unrefuted. Ignoring facts does not make 
> them go away.
> 
> Clearly, and we should choose the approach that maximizes the 
> return on our investment. Splitting capwap into two 
> sub-protocols to partially (and poorly) solve problems that 
> have already have lower cost solutions seems like a bad bet.
> 
> Scott
>

---

From: "Scott G. Kelly" <[email-masked]>
Date: Wed, 14 Jun 2006 13:24:10 -0400 (EDT)

Hi Pat,

>your points in various e-mails, not ignored them and hope they go away,

I agree that we're going in circles, but I dont agree that you've addressed each of my points. I posted a detailed cost/benefit analysis of two ports vs one port which clearly establishes the technical costs and deficiencies of the two port approach, but you never acknowledged, refuted, or commented on this.

>and the points raised on scaling haven't been refuted - or even

Am I missing something? The only scalability issues I've seen were in Rohit's post, and I thought it was quite clear that his concerns were based on the mistaken assumption that capwap data and control channels would share a dtls tunnel. This won't work, and has never been suggested (at least, not by me), so this should lay his concerns to rest.

Are there any other scalability concerns to be addressed?

>pain for long term interoperability issues.

I know you stated this yesterday, but (unless I've missed something) you have yet to substantiate this assertion. What would compel us to go down this path later?

>can be said at this point, other than to agree that we disagree.

Let's not personalize this by making the single-port proposal "mine" and the multi-port proposal "yours". The are options that are on the table, and should be objectively analyzed as such.

In that spirit, I remain willing to objectively review both options. However, the QoS arguments presented on behalf of the multi-port approach simply do not stand up to scrutiny, and the minimal benefits the multiport approach may provide in this regard are easily derived from other less costly approaches. This cannot be the only motivation for taking on so much implementation and operational pain - it defies reason.

Scott

---

From: "Bob O'Hara (boohara)" <[email-masked]>
Date: Wed, 14 Jun 2006 11:25:51 -0700

Scott, 

Perhaps separating the topics is a reasonable thing to do.  

Addressing your points below...

1. Characterizing the "vast majority" of hotspot deployments as local
MAC is incorrect.  The vast majority of hotspots fall outside the
architectures encompassed by CAPWAP.  They are classical, stand-alone
APs without any AC, at all.  Of those hotspot deployments that are not
stand-alone APs, there is a good mix of both local-MAC and split-MAC
hotspot deployments.  

2. The fact that users of today's hotspots have no expectation of QoS is
irrelevant to the discussion

3. The problem is real, regardless of whether you believe it to be, or
not.  An actual service provider has posted several more than once on
this list, saying exactly that.  Simply claiming that their problem is
irrelevant does not make it so.

4. The WTP is only in a unique position to take the actions you cite
(throttling down user traffic) only if the problem can be detected by
the WTP.  If the traffic is dropped beyond the link outbound from the
WTP, how does it learn what packets were dropped and where?  There is no
mechanism to provide this feedback to the WTP.

5. There are many implementation-specific features, such as those you
attribute to a potential local-MAC implementation, that are possible.
But, this strands the split-Mac architecture.

6. Being unable to imagine the problem does not invalidate it.  Again, a
service provider has validated exactly this problem and provided
specific support and examples for it.  The claim is that the WTP and AC
"*know*" that data flooding is causing loss of control channel packets.
The claim that this is so, is not proof of it.  Neither the AC nor the
WTP can know why any particular packet is dropped, or even if it was
dropped.  There is no feedback mechanism in CAPWAP providing such
feedback.  All that the WTP or AC know is that an expected packet did
not arrive, causing the state machine to change states.  There is no
ability to determine why that packet did not arrive.

7. A recurring point in this discussion, not only this email, is the
increased costs and complexity of the two port approach.  This is
incorrect.  There is no additional cost or complexity to this approach
in the AC or WTP implementation, even for NAT traversal.  NAT traversal
is required for even a single port.  If it can be accomplished for a
single port, it is no more difficult or costly to do it for one or more
additional ports.  The actual cost is in requiring
change/replacement/upgrading of intermediate networking equipment to
adapt to a new protocol, while providing the same services as are
provided to other protocols.

8. You state that the "vast majority" of deployments that might expect
reliable QoS are within a single domain of control.  Would you please
state the references of this data?

Many of the arguments presented below seem to be leading to a reduction
of the market to which the full capabilities of CAPWAP would apply.  I
believe that this is not appropriate.  In particular, the hotspot will
soon have expectations of QoS that are similar to those of an
enterprise.  The growth of VOIP on WLAN will drive the QoS requirement.
Designing a protocol, now, that does not encompass this requirement is
very short sighted.

 -Bob
 
-----Original Message-----
From: Scott G. Kelly [mailto:[email-masked]] 
Sent: Tuesday, June 13, 2006 5:27 PM
To: Bob O'Hara (boohara); capwap
Subject: reasonably bounding our QoS ambitions for the base protocol

I wanted to limit my earlier reply to the question the thread originally
was addressing (getting the premises straight), but I also think the
response below needs more attention. I've changed the subject line to
keep the two threads distinct.

Comments inline...

-----Original Message-----
>requiring any changes to the third party's network equipment.

Currently, the vast majority (if not all) of the deployed hotspots are
local MAC, and users of those hotspots have no expectation of QoS. If
the access point requires QoS for control traffic, then it also needs it
for the 802.11 mgmt traffic that it's forwarding to the AC, and the
hotspot provider would negotiate an appropriate agreement with the ISP -
but in this case, *both* the capwap control and data channels would
require QoS. However, there is a much simpler solution (assuming this
ever becomes a real problem).

If user data *were* choking the Internet link and hence causing control
traffic starvation, the WTP is in a unique position to detect this and
defend itself by throttling down user traffic while it attempts to
re-establish its control linkage. Also, there is no requirement that a
local-MAC WTP reset just because the control channel is temporarily lost
- indeed, isn't it common (in fact, demanded by customers) today for
local MAC WTP's to switch into a restricted mode upon loss of the
control channel connection, and to maintain existing user sessions while
rejecting new ones? That is, this particular problem is already being
effectively addressed in other ways.
 
>CAPWAP packet, the 5-tuple is the same for both control and data.

I'm having a really hard time with this one - forwarding user data back
to the AC in a hotspot scenario, and doing it over an untrusting (and
untrusted) 3rd party network, and expecting to provide QoS to boot? It's
hard to imagine this happening in practice, but the fact is that the
same argument applies with respect to control channel traffic, i.e. if a
flooded data channel is causing control channel starvation, the WTP or
AC *knows* this - it's immediately evident - and the WTP or AC can
throttle back the data channel flow in favor of the control channel. And
this requires no 3rd party intervention.

I'm sure that if we put our minds to it, we can dream up corner cases
which might not be addressed by a single port protocol,  but there are
other ways to solve the problems above that don't entail the complexity
and costs of the multiport approach. And as an aside, this would appear
to be one of the cases where NAT is most likely to occur, further
complicating things. 

----

I think it's important to acknowledge that none of the proposed
solutions is perfect, or will service all possible scenarios that we
might dream up. I think it's also important to realize that there are
many problems to be solved in wlan deployments, and we are not, indeed
cannot be chartered to solve each and every one of them here. We need to
solve a reasonable subset of them, and address other problems in later
efforts. 

The vast majority of real-world deployments where reliable provision of
QoS might be reasonably expected are within a single domain of control,
and those that are not present very challenging problems - problems
which should not within the charter of this working group at this time,
if we are to accomplish anything useful within a meaningful time frame. 

Scott

---

From: "Scott G. Kelly" <[email-masked]>
Date: Wed, 14 Jun 2006 14:10:25 -0700 (GMT-07:00)

Hi Bob,

>stand-alone APs, there is a good mix of both local-MAC and split-MAC

Of course, you are correct about the APs not being capwap wtp's. However, their current deployment model most closely reflects the intent of capwap's local mac model. Sorry for any confusion, and while strictly speaking you are correct, I don't think this invalidates any of the points that followed.

>irrelevant to the discussion

Actually, I think it *is* relevant, because it means that local (self-defense) throttling is a more reasonable solution to the problem, were it ever to occur (since there is no explicit user-data SLA to be concerned with). However, let's not get bogged down in this, because as you suggest, it's not central to the discussion

>irrelevant does not make it so.

I interpret his private communication to you and me as throwing this assertion into question, but it's up to him to post a public clarification if he cares to. That's all I can reasonably say on this point.

>mechanism to provide this feedback to the WTP.

If the WTP is not getting responses to control traffic, but it is seeing inbound data traffic, then it is immediately evident that the internet connection is up. If the link is saturated, throttling it down and trying again with control traffic is a viable (and advisable) adaptation. Of course, many things could go wrong with WTP-AC link further up the line, but the argument this started with is that we need differential QoS between control and data to prevent data flooding from causing control traffic starvation. The WTP is clearly in a position to detect and mitigate this.

>But, this strands the split-Mac architecture.

Again, we must keep in mind the problem we are purporting to solve with two ports: differential control vs data channel QoS.

Because the WTP can detect and throttle the flooding data flow, I don't see how split-mac architectures are stranded.

>specific support and examples for it.  

See response to 3, above.

>The claim is that the WTP and AC

No, the claim is that they know when the two are occurring together, and they can then attempt to prove the two are correlated quite simply. Is it absolute? No. Is the two-port approach absolute? No.

>ability to determine why that packet did not arrive.

See above.

>provided to other protocols.

See http://lists.frascone.com/pipermail/capwap/msg02798.html

Here's the reader's digest version: we get nat-t for free with the single port approach, due to the built-in heartbeat. It's clearly much more expensive with the multi-port approach.

>state the references of this data?

I don't have time to research and prove this, and I concede that I am speaking from experience.

>Designing a protocol, now, that does not encompass this requirement is

I agree that some of the arguments do lead us to a reduction of the functionality we are trying to provide in rev 1 of a base protocol, but I don't think this is bad. I think we must limit our ambitions for the first rev if we are to get something out there soon.

The IETF has seen a number of attempts at swiss-army-knife-style specification, and they tend to fail miserably.  So, it seems prudent to set our sights a little lower at first, being careful not to slam doors shut, but at the same time not trying to solve everything conceivable in the first pass. That's one of the reasons we have version numbers in these protocols.

It is reasonable to try to anticipate likely developments and try to anticipate them in the protocol design, and VoIP (and QoS in general) is clearly very important. But VoIP, video, and other QoS drivers are not being addressed by either of these proposals. 

Scott