Subject: [Capwap] Some transport comments on the CAPWAP protocol

From: Magnus Westerlund <[email-masked]>
Date: Tue, 10 Jul 2007 17:07:05 +0200

Hi,

Sorry for having been very tardy in sending these comments to the WG. I 
had a discussion with the WG chairs back in April. But I totally forgot 
to summarize what issues I saw and for which wasn't directly apparent 
being me missunderstanding things. I do hope we can ensure that that 
these are quickly resolved. I also hope that your transport advisor 
David Borman will be able to work more actively with you to sort things 
out.

One comment I had, was in regards to the lock-step protocol you have 
defined for transfer of firmware. First of all I wished you back when 
you started this year would have made it simpler for you by using a 
off-the-shelf transport protocol suitable for bulk transfer. But as 
these decisions where made back in history lets just ensure that your 
protocol has considered all the neccesary issues. I do hope that it will 
deliver enough performance for you to be a workable solution. Therefore, 
I do have the question on what range of RTT you expect to have between 
the AC and the WTP? This as the RTT has major performance impact.

My next question was if the protocol addresses all the issues raised in 
the security considerations for TFTP, which after all is a similar 
protocol in regards to the firmware transfer functions. So can someone 
please enlighten me by answering in regards to these things raised in 
section 5 of RFC 3617:

    Use of TFTP has been historically limited to those devices where a
    more full protocol stack is impractical due to either memory or CPU
    constraints.  While this still may be the case with a toaster, it is
    unlikely to be the case for even the simplest piece of network
    support hardware, such as simple routers or switches.  There are a
    myriad of reasons to use some protocol other than TFTP, only a few of
    which are listed below.

    TFTP has no mechanism for access control within the protocol, and
    there is no protection from a man in the middle attack.
    Implementations are left to their own devices in this area.  Because
    TFTP has no way to determine file sizes in advance, implementations
    should be prepared to properly check the bounds of transfers so that
    neither memory nor disk limitations are exceeded.

    TFTP is not well suited to large files for the following reasons.
    TFTP has no inherent integrity check.  There is no way to determine
    what one side sent is what the other received.  There is no way to
    restart TFTP transfers from anywhere other than the beginning.  TFTP
    is a lock step protocol.  Only one packet may be in flight at any one
    time.  There is no slow start or smart backoff mechanism in TFTP, but
    very simple timeouts.

    TFTP is not well suited to file transfers across administrative
    domains.  For one thing, TFTP utilizes UDP, and many NATs will not
    either support or allow TFTP transfers.  More likely firewalls will
    prohibit transfers.

    There are no caching semantics within TFTP.  There is no safe way to
    cache information using the TFTP protocol.

The next comment was to ensure that you have in your protocol considered 
all the issues raised by the transport area's UDP guidelines document 
that we are developing. Please consider the issues mentioned and we 
greately appreciate any feedback on the document itself.
http://www.ietf.org/internet-drafts/draft-ietf-tsvwg-udp-guidelines-02.txt

I also raised a comment about you specifying that the checksum is turned 
off for UDP. I wonder if there are any real reason for doing this? I did 
understand that the usage of DTLS is all the integrity checks you need 
for the individual packets. However, I would like you to consider the 
fact that for IPv6 you will be required to have it on. Also it prevents 
any port/device receiving a packet that gets a bit-error in the packet 
to discard it as the obvious stray, or forces DTLS to check it when the 
"for free" UDP checksum would take care of it.

Best Regards

Magnus Westerlund

IETF Transport Area Director & TSVWG Chair
----------------------------------------------------------------------
Multimedia Technologies, Ericsson Research EAB/TVM/M
----------------------------------------------------------------------
Ericsson AB                | Phone +46 8 4048287
Torshamsgatan 23           | Fax   +46 8 7575550
S-164 80 Stockholm, Sweden | mailto: [email-masked]
----------------------------------------------------------------------
_________________________________________________________________
To unsubscribe or modify your subscription options, please visit:
http://lists.frascone.com/mailman/listinfo/capwap

Archives: http://lists.frascone.com/pipermail/capwap

---

From: tlais <[email-masked]>
Date: Thu, 12 Jul 2007 16:45:47 +0200

Hi all,

I am working, in my PhD thesis, on the centralized WLAN architecture
based on access controllers discussed in the taxonomy RFC. I am wondering
if the access controller entity is able to reduce and/or increase the 
wireless
bandwidth allocated to mobile terminals.

many thanks for your help

Regards
Mazen Tlais

---

From: Margaret Wasserman <[email-masked]>
Date: Wed, 18 Jul 2007 04:41:44 -0400

Hi Magnus,

There hasn't been any discussion of these issues, so I thought I'd  
try to kick some off...

I'd first like to thank you for raising these issues with the group  
now, rather than waiting for the IESG review period.  This is very  
helpful, as it will allow us to resolve the issues earlier and  
present a better document to the IESG.  Thanks!

I noticed that you didn't include anything explicit about MTU  
discovery in this message, but I know that issues was raised by the  
Transport area, perhaps by Lars?   Specifically, I remember a  
discussion that concluded that CAPWAP would need to have a mechanism  
to decide what MTU to use and would need to default to a minimum MTU  
size (512 bytes?) when no other information was available, in order  
to avoid fragmentation.  Is that still a concern regarding the CAPWAP  
protocol?  If so, do you (or Dave or Lars) have any advice about how  
we might address it?

The following are just my personal thoughts and opinions, not  
anything official as a WG chair.

<wgchair hat=off>

I've looked over your comments, and I think that CAPWAP properly  
addresses most or all of the issues that you've raised.  I've  
included explanations in-line below, and I'd be interested to know if  
any of you continue to see any problems that need to be addressed  
after reading my explanations.

On Jul 10, 2007, at 11:07 AM, Magnus Westerlund wrote:

> Therefore,

Today, the main way that WTPs would get firmware is to download it  
using TFTP at boot time before the device becomes fully operational.   
CAPWAP is intended to allow more dynamic firmware management (so  
download can happen while the device is operational), which will  
allow for "trickle firmware download", thus reducing the need for  
performance of  firmware downloads.

So, performance of firmware downloads is not something we've been  
particularly concerned about in the CAPWAP context.  Is there some  
reason that you believe that we should be concerned about this?

>     Implementations are left to their own devices in this area.

In CAPWAP, the firmware download happens over the control channel,  
which is always an authenticated and encrypted DTLS connection.

>    Because

In CAPWAP, we do send the size of the file prior to transferring the  
data.

>     what one side sent is what the other received.

The DTLS connection provides for data integrity.  Also, the  
acknowledgment mechanism in CAPWAP allows the AC to know whether the  
WTP has received a specific block.  The AC will not send another  
block until the previous one is received.

>    There is no way to
>     restart TFTP transfers from anywhere other than the beginning.

This is true of the CAPWAP model, but I don't know why it would be a  
problem in practice.

>     TFTP

CAPWAP is also a lock step protocol, but the CAPWAP retransmission  
mechanism includes exponential back-off, doubling the wait time  
between each retransmission.

>     domains.

Since the CAPWAP firmware download mechanism uses the same  
authentication and transport as the CAPWAP control channel, it is  
just as suited to use across administrative domains as CAPWAP.  It  
isn't that clear to me, personally, that CAPWAP will be widely used  
across administrative domains, but I don't believe that the firmware  
download mechanisms presents any restrictions in this area.

>     cache information using the TFTP protocol.

I am not sure how/if caching could/should apply in CAPWAP.  Each AC  
has its own direct connection to the WTP.  In many cases, an AC will  
have one or more local firmware images available and will not re- 
download an image that it already has.  However, CAPWAP also supports  
devices that don't hold any static image and always download an image  
when they boot.

> http://www.ietf.org/internet-drafts/draft-ietf-tsvwg-udp- 

I think that CAPWAP properly covers the "Congestion Control",  
"Reliability Guidelines" and "Checksum Guidelines" described in this  
document.  If you have any remaining concerns in this area after  
reading this response, please let me know so that we can discuss them  
further.

As I mentioned above, I think we may need to address MTU discovery  
issues within CAPWAP, and we could probably use some advice from the  
transport area on how best to do that.  Could we piggyback on an  
existing MTU discovery mechanism, like Path MTUD?  Can you give us an  
example of a UDP-based protocol that handles this well, so that we  
could use it as a guide?

I did not understand the "Middlebox Traversal Guidelines" section of  
this document well enough to understand how/if it applies to CAPWAP.   
Do you have any specific concerns in that area?

> "for free" UDP checksum would take care of it.

This issue was explicitly discussed by the group, and, for  
performance reasons, we chose to use UDP-Lite for IPv4 CAPWAP  
connections, so that the UDP checksum will cover only the first 8  
bytes of the packet (the UDP header) and not the UDP data, which will  
be processed via DTLS.   We do understand that we will be required to  
use UDP with checksums enabled for IPv6.   This area of the document  
was fairly recently updated to discuss UDP-Lite, so please let us  
know if you have any remaining concerns in this area.

</wgchair>

Thanks again for raising these issues with the group.

Margaret

---

From: Magnus Westerlund <[email-masked]>
Date: Wed, 18 Jul 2007 11:45:05 +0200

Hi,

See inline. If people are going to discuss the potential and open issues 
issues, I would strongly suggest that the split each issue into a 
separate thread. However, I don't like to do that as I am commenting and 
confirming all of the raised points.

But to me it appears that the potential issues remaining are:

- Download performance
- MTU discovery
- Middleboxes
- UDP checksums

Margaret Wasserman skrev:
> 
> Hi Magnus,
> 
> There hasn't been any discussion of these issues, so I thought I'd try 
> to kick some off...
> 
> I'd first like to thank you for raising these issues with the group now, 
> rather than waiting for the IESG review period.  This is very helpful, 
> as it will allow us to resolve the issues earlier and present a better 
> document to the IESG.  Thanks!
> 
> I noticed that you didn't include anything explicit about MTU discovery 
> in this message, but I know that issues was raised by the Transport 
> area, perhaps by Lars?   Specifically, I remember a discussion that 
> concluded that CAPWAP would need to have a mechanism to decide what MTU 
> to use and would need to default to a minimum MTU size (512 bytes?) when 
> no other information was available, in order to avoid fragmentation.  Is 
> that still a concern regarding the CAPWAP protocol?  If so, do you (or 
> Dave or Lars) have any advice about how we might address it?

Yes, you likely need to have a MTU discovery mechanism to be avoid 
needing to use very conservative packet sizes. But there should be 
possible to implement something like the MTU discovery method described 
in RFC 4821 in the CAPWAP protocol.

> 
> The following are just my personal thoughts and opinions, not anything 
> official as a WG chair.
> 
> <wgchair hat=off>
> 
> addresses most or all of the issues that you've raised.  I've included 
> explanations in-line below, and I'd be interested to know if any of you 
> continue to see any problems that need to be addressed after reading my 
> explanations.
> 
> Today, the main way that WTPs would get firmware is to download it using 
> TFTP at boot time before the device becomes fully operational.  CAPWAP 
> is intended to allow more dynamic firmware management (so download can 
> happen while the device is operational), which will allow for "trickle 
> firmware download", thus reducing the need for performance of  firmware 
> downloads.
> 

I am lacking the information to make a good judgement of this. What size 
are the firmware downloads that going to happen. And what RTTs are 
expected. I think one can make a reasonable worst time estimate of the 
download time considering a scenario where firmware has grown at least 
10 times and the RTT is twice what is expected. Include 5% packet loss 
(taking retransmission timers into account) and use an MTU of 512 bytes 
and condiering all overhead and then calculate how long it would take to 
complete. If one is fine with the time that is expected to take then I 
think we are reasonably safe also a bit into the future.

I would really appreciate if someone took the time to do the calculation 
and reported back to the list.

> In CAPWAP, the firmware download happens over the control channel, which 
> is always an authenticated and encrypted DTLS connection.

Okay, not an issue.

> In CAPWAP, we do send the size of the file prior to transferring the data.

Okay, not an issue.

> acknowledgment mechanism in CAPWAP allows the AC to know whether the WTP 
> has received a specific block.  The AC will not send another block until 
> the previous one is received.

This is not an issues, as you have both per packet checks and hash over 
the complete file, also capturing reassembly errors.

> This is true of the CAPWAP model, but I don't know why it would be a 

Not, as long as the downloads complete in reasonable timeframes. For 
downloads taking a long time where there is potential for interruption 
in either server or connection this could be an issue.

> mechanism includes exponential back-off, doubling the wait time between 
> each retransmission.
> 

Yes, CAPWAP appears to me to fulfill all the congestion control 
requirements we (TSV area) sees needed.

> authentication and transport as the CAPWAP control channel, it is just 
> as suited to use across administrative domains as CAPWAP.  It isn't that 
> clear to me, personally, that CAPWAP will be widely used across 
> administrative domains, but I don't believe that the firmware download 
> mechanisms presents any restrictions in this area.

No, as you have authentication etc. This should not be an issue.

> I am not sure how/if caching could/should apply in CAPWAP.  Each AC has 
> its own direct connection to the WTP.  In many cases, an AC will have 
> one or more local firmware images available and will not re-download an 
> image that it already has.  However, CAPWAP also supports devices that 
> don't hold any static image and always download an image when they boot.

No, I think your limited usage of the download capabilites does not 
require caching mechanism. I think this point in regards to TFTP is its 
usage as a more general file delivery mechanism, like FTP or HTTP. In 
which case caching makes more sense to include in the architecture.

> document.  If you have any remaining concerns in this area after reading 
> this response, please let me know so that we can discuss them further.

Okay, I personally don't think so. But I do hope Dave will have reviewed 
your spec before the meeting and can provide any further insights into 
this.

> example of a UDP-based protocol that handles this well, so that we could 
> use it as a guide?

I think the easiest may be to use RFC 4821 style incorporated into the 
protocol if possible. That allows you to allways utilized the available 
MTU well. I don't see using classicla ICMP based will really provide a 
great benefit other than making things more complex and more vunerable.

I think you have most of the tool necessary for RFC 4821 style checks in 
the protocol already in the ECHO request/response + MTU discovery 
padding message. Only some procedural text may need to be written.

> this document well enough to understand how/if it applies to CAPWAP.  Do 
> you have any specific concerns in that area?

A question to the group is if anyone really thinks that there will be 
any middleboxes (NATs or firewalls) in the path between the STA, WTP or 
AC. To me it appears that it is more likely that these are located 
outside of this system but I may be wrong.

Also this should not be an issue as long as the node on the inside 
initiate all bi-directonal message sequences and the peer responds back 
to the source of the messages. The real issue with middleboxes are 
usually when an external peer opens a new packet flow towards the inside 
of an middlebox. If you don't have these then there should be minimal 
concern.

> This issue was explicitly discussed by the group, and, for performance 
> reasons, we chose to use UDP-Lite for IPv4 CAPWAP connections, so that 
> the UDP checksum will cover only the first 8 bytes of the packet (the 
> UDP header) and not the UDP data, which will be processed via DTLS.   We 
> do understand that we will be required to use UDP with checksums enabled 
> for IPv6.   This area of the document was fairly recently updated to 
> discuss UDP-Lite, so please let us know if you have any remaining 
> concerns in this area.
> 

You have had the discussion. Can someone please recap what arguments was 
used to indicate that you would have better performance by not using the 
UDP checksum over the payload? I was under the impression that unless 
your capwap boxes are running very special hardware the UDP checksuming 
was done in hardware on the network controler. Thus there is no cost for 
doing the checksum. And it would save the DTLS processing in the case 
the packet has a bad checksum.

I am trying to understand your argumentation here, not necessarily 
change it.

Cheers

Magnus Westerlund

---

From: Margaret Wasserman <[email-masked]>
Date: Wed, 18 Jul 2007 10:06:45 -0400

Hi Magnus,

Thanks for the quick response!

> But to me it appears that the potential issues remaining are:

I agree that we need to discuss each of these issues further, and  
that we should do so in separate threads.

I'll open a tracker issue for each of these topics (using the  
information from this thread) and I'll send separate messages to the  
list containing each issue number and summarizing each issue.   
Hopefully we can have an organized discussion in those threads.

Margaret

---

From: "Pat Calhoun (pacalhou)" <[email-masked]>
Date: Mon, 23 Jul 2007 07:14:11 -0700

My apologies for the latency in my response, but here are my 2 cents on
the issues raised.

> One comment I had, was in regards to the lock-step protocol you have
defined for transfer of 
> firmware. First of all I wished you back when you started this year
would have made it simpler for 
> you by using a off-the-shelf transport protocol suitable for bulk
transfer. But as these decisions 
> where made back in history lets just ensure that your protocol has
considered all the neccesary 
> issues. I do hope that it will deliver enough performance for you to
be a workable solution. 
> Therefore, I do have the question on what range of RTT you expect to
have between the AC and the WTP? 
> This as the RTT has major performance impact.
My current experience is that the majority of WTPs will be deployed in
the enterprise. These enterprise networks are typically over-provisioned
and RTT is very low (sub 10ms). That said, there are some deployments of
WTPs in branch offices, connected either via a private or public network
to the campus, where RTT can range up to 10-100ms (where the high end is
typically rare as it impacts other enterprise applications). 

> My next question was if the protocol addresses all the issues raised
in the security considerations 
> for TFTP, which after all is a similar protocol in regards to the
firmware transfer functions. So can 
> someone please enlighten me by answering in regards to these things
>     myriad of reasons to use some protocol other than TFTP, only a few
of
Well, the good thing is that most govenmental bodies prevents the
transmit power on most APs, limiting their use as toasters.

Because
>     neither memory nor disk limitations are exceeded.

So there are a couple of issues listed here, so I will tackle them one
at a time.
1. Man in the Middle. This is less of an issue with the file transfer
and is more of a question as to whether the CAPWAP protocol is
susceptible to Man in the Middle attacks. Please note that CAPWAP uses
the DTLS protocol, and therefore leverages from all of the security
characteristics that protocol provides. Further, the CAPWAP specifically
includes addition authorization of peers, where it takes the credentials
exchanged during the key exchange, and validates that the peer is an
authorized CAPWAP peer.
2. File Size. During the download phase, the Image Information message
element is included in the Image Data Response command, which includes
the file size. In the CAPWAP protocol, the image is transferred from the
AC to the WTP, and the AC provides this information to the WTP. However,
the WTP initiates the image transfer, so if it had determined it did not
have sufficient storage space, it would simply skip the image transfer.
I would point you to section 9 of the current spec which includes
diagrams depicting the message exchanges and information included.

TFTP
>     time.  There is no slow start or smart backoff mechanism in TFTP,
but
Correct. I would note that first off, we are not using TFTP so many of
these issues are not inherent in the CAPWAP protocol. So let me talk
about these one at a time.
1. Lock Step. The CAPWAP Control Protocol is in fact a lock step
protocol, and therefore only one message is in flight at any given time.
The AC sends a portion of the image, which the WTP acknowledges. This
causes the next block to be transferred. There is no need to go back to
the beginning, because each block is checksum'ed (see next sub-point),
so if it were corrupted, the WTP would response with a negative response
indicating the packet was invalid (or it simply does not respond, which
causes a retry). Given this is used to download firmware to the WTP, and
this can be done while the WTP is providing service, I do not see this
as much of an issue.
2. No Checksum. As late as the -05 draft, the CAPWAP Image Data message
element included a checksum. However, there was a request from the
working group to eliminate this, which had rough consensus, for the
following reason:
    > 6) The Image Data Message element has several problems. This
    >     include:
    >       1) it has a checksum that is used to determine if the
    >          block of image data has been modified. However, the
    >          DTLS session provides information to determine if
    >          a message has been modified. Also, the algorithm
    >          for the checksum is not specified.
    >          (This can be resolved by removing from each message
    >          element, and providing for a digest, such as MD5,
    >          for the entire file.)
Subsequently, the checksum field was removed, and draft -06 no longer
included this field.
3. Exponential backoffs. Support for exponential backoffs was included
in the draft -06, and was tracked as issue 251. The resulting text is:
    4.4.3.  Retransmissions
    [...]
       After transmitting a Request message, the RetransmitInterval (see
       Section 4.6) timer and MaxRetransmit (see Section 4.7) variable
are
       used to determine if the original Request message needs to be
       retransmitted.  The RetransmitInterval timer is used the first
time
       the Request is retransmitted.  The timer is then doubled every
       subsequent time the same Request message is retransmitted, up to
       MaxRetransmit but no more than half the EchoInterval timer (see
       Section 4.6.5).  Response messages are not subject to these
timers.

>     either support or allow TFTP transfers.  More likely firewalls
will
Honestly, I think NATs these days are perfectly capable of transporting
UDP, but with some restrictions. Obviously, they use timers to set ports
to the idle state. It is for this reason that the CAPWAP protocol
includes a keepalive mechanism, which ensures that the NAT will keep the
UDP binding alive. Note that this pertains to the CAPWAP protocol as a
whole, and not simply the image download mechanism.

>     cache information using the TFTP protocol.
I do not understand what this means, sorry.

> The next comment was to ensure that you have in your protocol
considered all the issues raised by the 
> transport area's UDP guidelines document that we are developing.
Please consider the issues mentioned 
> and we greately appreciate any feedback on the document itself.
>
http://www.ietf.org/internet-drafts/draft-ietf-tsvwg-udp-guidelines-02.t
xt

Ok, well there's a myriad of issues here so I guess I will have to
address each one in turn.
3.1 Congestion Control
As described above, the CAPWAP control plane does include exponential
backoff, which provides congestion control. However, the CAPWAP data
plane does not include congestion control, as it is simply a tunnel
between the WTP and the AC, similar to GRE. I believe that attempting to
add congestion control to the data plane would introduce another set of
issues and given the overwhleming majority of CAPWAP deployments simply
does not cause a need for it.

3.2 Message Size Guidelines
Well... This is an interesting guideline, and I question why it exists.
As background information, the CAPWAP protocol was based on LWAPP, and
it shares the same PMTU of 1468. This is basically the Ethernet MTU
minus the various headers. I would note that today, there are well over
4M WTPs that run the LWAPP protocol, connected everywhere from the
campus to branch to home office. The only issue we had ever seen was
with PPPoE, which required us to reduce our MTU size. We have customers
using all kinds of private and public networks, and this has never been
a problem for any one of our deployments. 

So while I am sure that the authors of the draft must have had a really
good reason for recommending the message size, I question why it never
caused any issues for any of our LWAPP deployments.

3.3.  Reliability Guidelines
The CAPWAP control protocol has inherent reliability through a
request/response scheme. It also ensures that the messages are
identified to ensure that any retransmissions are identified, and there
is specific text around the treatment of the retransmissions. The data
plan does not include reliability.

3.4 Checksum Guidelines
This is discussed below as a separate point.

3.5 Middlebox Guidelines
As discussed above, the CAPWAP protocol includes a keepalive mechanism,
which is sent periodically to ensure that the UDP flow is kept alive.
Furthermore, in order to ensure that the data plane UDP session is kept
alive at the NAT, a keepalive mechanism also exists in the data plane. I
note that the guidelines state that this should be sent no more frequent
that every two minutes, which in my personal experience is on the high
side. The CAPWAP protocol uses a configurable timer, whose default value
is 30 seconds, which allows us to support the realities of life because
there are middleboxes out there that will in fact violate the 2 minute
recommendation (and as with any vendor, interoperability with the
network and reducing support calls is key).

> I also raised a comment about you specifying that the checksum is
turned off for UDP. I wonder if 
> there are any real reason for doing this? I did understand that the
usage of DTLS is all the 
> integrity checks you need for the individual packets. However, I would
like you to consider the fact 
> that for IPv6 you will be required to have it on. Also it prevents any
port/device receiving a packet 
> that gets a bit-error in the packet to discard it as the obvious
stray, or forces DTLS to check it 
> when the "for free" UDP checksum would take care of it.
There are two types of packets in CAPWAP; control and data. With very
few exceptions, which are packets sent prior to the establishment of the
DTLS session, the control packets will be checksum'ed as a result of
DTLS. However, for data packets this is not the case, as the use of DTLS
on the data path is optional.

Recall that the primary function of the AC on the data plane is to
convert packets from a CAPWAP tunneled packet to the 802.3 format. Most
ACs designed today need to do this at very fast speeds, and in many
cases this is all done in hardware. When we discussed this during the
protocol design, there was overwhelming consensus that needing to run
checksums in hardware at high speeds was expensive, which led to the
disabling of checksum. 

Again, for vendors that care, they would use DTLS, which in itself
significantly increases the cost of the hardware. To your point on IPv6,
it is for this reason that we recommended the use of UDP-Lite for IPv6.

PatC