Subject: [tsv-area] ICE pacing connectivity checks

From: "Dan Wing" <[email-masked]>
Date: Thu, 26 Apr 2007 08:54:02 -0700

Lars and Magnus raised an issue in MMUSIC that ICE's timers may not be TCP
friendly
<http://www1.ietf.org/mail-archive/web/mmusic/current/msg05571.html>.  We
are moving this discussion to TSVWG where there is better interest in this
specific matter.

Background:
-----------

ICE is http://tools.ietf.org/html/draft-ietf-mmusic-ice-15, and ICE's
primary purpose is for NAT traversal of RTP media flows.  To do this, ICE
gathers a list of candidate addresses from both media endpoints (private
IPv4 address, public IPv4 address, IPv6 addresses, VPN'd addresses, and so
on), exchange those addresses with each other, and then performs
connectivity checks.  A more detailed explanation of the steps is the
Overview section of the ICE specification.

Details:
--------

The primary concern is that ICE's connectivity checks are worse to the
network than an RTP flow with RTCP Feedback (RFC4585).  RFC4585 allows an
RTP receiver to issue an RTCP message after 500ms, and that message can tell
the RTP sender to 'slow down'; this message might mean switch to a more
bandwidth-efficient codec, for example.  For the first 500ms, however, the
sender will send RTP packets towards the receiver.

The current ICE specification (ice-15) uses two timers to control its
pacing; one is "Ta" which controls when the first connectivity check is sent
for each candidate address and is 20ms.  The other is STUN's retransmit
interval which is defined in draft-ietf-behave-rfc3489bis-06 to start at
100ms and double thereafter (100ms, 300ms, 700ms, 1500ms, 3100ms, 6300ms).

I was able to model ICE with two a=candidate lines (4 candidate pairs) and
STUN's currently-specified timers are modeled against an RTP payload size of
40 bytes and RTP "ptime" (packetization interval) of 20ms, ICE uses more
bandwidth for the first 440ms, at which time RTP consumes more bandwidth.
If we assume both endpoints use RTCP Feedback (RFC4585), in approximately
another 60ms the sender's RTP will be informed of packet loss and slow down.

Of course, if any of ICE's connectivity checks complete (that is, we receive
a STUN Response), ICE processing stops and soon thereafter RTP would start.
This, however, is difficult to model and I did not attempt to model this.

With more conservative RTP payload sizes or RTP packetization intervals, or
with a lot more candidate pairs, ICE becomes quite a lot worse than RTP.  

Proposal:
---------

I propose the following formulas which prevent ICE from exceeding the
bandwidth RTP would have used:

  Initial STUN transaction delay ("Ta") = 
     stun_packet_size / rtp_packet_size * rtp_ptime

  STUN retransmit interval ("RTO") = 
    Ta * candidate_count

These formulas cause us to first send a STUN Binding Request to each
a=candidate, at a rate that consumes the same bandwidth as the RTP flow we
will eventually be establishing with the remote endpoint.  If we don't get
any responses we then start retransmitting with a 2x backoff.

I have modeled these formulas in the Excel spreadsheet at
http://www.employees.org/~dwing/ice-rtp-backoff.xls, where you can also
input Ta=20ms and RTO=100ms and various numbers of candidate pairs to see
how ICE compares with RTP.  The yellow fields are where you can make
adjustments without breaking the spreadsheet.  You can see the bandwidth
used by RTP (including IP, UDP, and PPPoE overhead) versus ICE in the middle
of the spreadsheet, as time increases from 0ms to 1000ms on the left.  The
second worksheet in that workbook has a graph of the same data.  

The advantage of this proposal is that ICE's bandwidth consumption is the
same as RTP's bandwidth consumption.  The disadvantage is that packet loss
of the ICE connectivity checks will likely cause us to learn more slowly
that we do, in fact, have connectivity with a more-preferred candidate.  Of
course, injecting packets into the network faster than the network expected
would, itself, be likely to contribute to packet loss causing the same
situation.  Being kind to the network seems the best approach.

Please let me know if this approach seems reasonable.

-d

---

From: Magnus Westerlund <[email-masked]>
Date: Thu, 26 Apr 2007 18:56:07 +0200

Hi

A few clarifications

> are moving this discussion to TSVWG where there is better interest in this

Our general concern is that ICE has no congestion control and in worst 
case can consume large amounts of bandwidth. The primary desire would be 
to have ICE automatically adopt to the amount of available bandwidth on 
the different paths. However that is almost impossible as there is no 
information if the connectivity check is lost due to reaching a NAT 
without an active binding, its filtering behavior or congestion 
introduced loss. Making this into a very hard problem.

I made a comparison with RTP when using RTCP as at least being able to 
react to congestion on a medium timescale. Not TCP-friendly but at least 
it prevents congestion collapse. Also RTP flows can range from 1kb up to 
a couple of GB so there is no good comparison to do. Thus I would like 
to steer away from comparing with RTP as being good or bad.

[snip]

> with a lot more candidate pairs, ICE becomes quite a lot worse than RTP.  

I am especially worried about the increase in number of candidate pairs 
as this is what makes the bandwidth run away as it currently is defined. 
What I think is important for the proposals is that the maintain a 
sensible bandwidth for the whole connectivity check phase independent of 
number of candidate on any number of streams, checkboards etc.

I would also ask people to consider if there is anything that can be 
improved compared relatively to use some basic RTP bandwidth for the 
planned stream.

There are contradicting requirements here. On one side avoid causing 
serious congestion issues, on the other have a reasonable time for 
establishing working paths for all the media streams.

Cheers

Magnus Westerlund

IETF Transport Area Director & TSVWG Chair
----------------------------------------------------------------------
Multimedia Technologies, Ericsson Research EAB/TVM/M
----------------------------------------------------------------------
Ericsson AB                | Phone +46 8 4048287
Torshamsgatan 23           | Fax   +46 8 7575550
S-164 80 Stockholm, Sweden | mailto: [email-masked]
----------------------------------------------------------------------

---

From: Sally Floyd <[email-masked]>
Date: Thu, 26 Apr 2007 15:29:24 -0700

> case can consume large amounts of bandwidth. The primary desire would 
> be to have ICE automatically adopt to the amount of available 
> bandwidth on the different paths. However that is almost impossible as 
> there is no information if the connectivity check is lost due to 
> reaching a NAT without an active binding, its filtering behavior or 
> congestion introduced loss. Making this into a very hard problem.
>
> react to congestion on a medium timescale. Not TCP-friendly but at 
> least it prevents congestion collapse. Also RTP flows can range from 
> 1kb up to a couple of GB so there is no good comparison to do. Thus I 
> would like to steer away from comparing with RTP as being good or bad.

...
> I am especially worried about the increase in number of candidate 
> pairs as this is what makes the bandwidth run away as it currently is 
> defined. What I think is important for the proposals is that the 
> maintain a sensible bandwidth for the whole connectivity check phase 
> independent of number of candidate on any number of streams, 
> checkboards etc.

I would agree with this.  (I was assigned a few days ago to review
the ICE draft for congestion control issues, and so far I have read
the relevant parts once-over-lightly, so I am not coming in
with any particular expertise in this.)

A first step would be to have some text like Section 17.4.2 on the
STUN Amplification Attack.  First for the conformant case (no one
misbehaving), with a worst-case ICE scenario with many candidate
pairs and such, describing the amount of bandwidth that ICE could
easily end up using, and the time interval that this bandwidth could
be used for.

And then a mention of the security issues already discussed, or
other issues if there are any, of things going wrong that could
make the bandwidth used much worse.  Just for completeness.

Given this situation where loss cannot be used as an indication of
congestion, and where the "round-trip time" isn't really a meaningful
concept, controlling the delay Ta and the STUN retransmit interval
RTO, from Dan's email, certainly seem like the right knobs to play
with.  But instead of allowing ICE to use at most the bandwidth
RTP would have used for the first 500 ms, I would suggest allowing
ICE to use at most X Kbps, for some value of X.  50, 100, 200 Kbps?
I don't know.

When the appropriate guidelines for Ta and RTO have been figured out,
I would recommend saying that "Y" SHOULD be the value or equation used,
whatever "Y" turns out to be, but that ICE MUST NOT use a value
more aggressive than Y.  So that some hosts might know
that they want to go slower than the recommended approach, but
that they MAY NOT go faster.  (Or they MAY NOT go faster unless
some really explicit conditions have been fulfulled.)

As a comparison of how bad TCP can be, after the SYN/ACK exchange
TCP can send an initial window of roughly 4K bytes/RTT (~32 Kbits/RTT)
(from RFC 3390).  And because presumably end nodes won't be starting
ICE sessions as frequently as they start TCP connections, it would
seem reasonable to me to allow ICE to be more aggressive that TCP,
in terms of the bandwidth used in the short initial period roughly
equivalent to an RTT.  (And if there was some way to *ensure* that
end nodes didn't open many concurrent or subsequent ICE sessions,
this argument would be even more convincing...)

Another issue in ICE's favor is that it seems acceptable (to me)
to evaluate the bandwidth used by ICE in terms of bytes per second,
not in terms of packets per second.  E.g., RFC 3714, RFC 4828.
(This is based on the assumption, which is not necessarily always
correct, that the bottleneck is likely to be the bandwidth in
bits per second, not the CPU processing in packets per second.)

Take these as first thoughts...

- Sally
http://www.icir.org/floyd/

---

From: Sally Floyd <[email-masked]>
Date: Thu, 26 Apr 2007 16:26:13 -0700

Another thought is that congestion control for ICE could be made
somewhat more intelligent if ECN was used, and the router for the
congested link was ECN-Capable.  In this case, ICE packets could be
sent as ECN-capable, and if an end-node received an ICE packet
that was ECN-marked, that end-node would tell the other end, and
the other end would know that the path was congested.

(If the ECN-Capable ICE sending end-node heard nothing, on the other
hand, it could not use this as an indication that the path was
uncongested; even if the end-node has some way to know that all the
routers in its universe are ECN-capabke, an ECN-capable router can
still have a buffer overflow and can drop packets due to congestion.)

I am not suggested this as something to figure out and add to the
current draft - it could take a while to figure out if it would or
would not work, and it could take some time.  But if some of the
ICE people were ever motivated to take it on and add it in a
new internet-draft, it might be worth the trouble...

(I would note that using ECN for ICE would always be a bit problematic.
For TCP, if an ECN-marked data packet is dropped downstream, that
is taken by TCP as an indication of congestion, so when a packet
is ECN-marked, that indication of congestion makes its way back to
the sending node one way or anoother.  In contrast, with ICE a
packet could be ECN-marked, and then dropped downstream, and this
would not be taken by the end nodes as an indication of congestion.
 From the router's point of view, this might not be a strong
recommendation for using ECN for ICE packets.  But I think it could
be argued that the tradeoffs would make it worth it, all in all.)

- Sally

---

From: Magnus Westerlund <[email-masked]>
Date: Fri, 27 Apr 2007 16:01:18 +0200

> be argued that the tradeoffs would make it worth it, all in all.)

The problem with using ECN is that most of the sent connectivity checks 
will be dropped by the external side of NATs or hosts that don't have an 
open port or for which the arrived packet is garbage and discarded. Only 
on packets where STUN requests are answered are there any possibilities 
to use ECN to notice that there are some issues on the path.

Cheers

Magnus Westerlund

---

From: "Dan Wing" <[email-masked]>
Date: Fri, 27 Apr 2007 08:42:20 -0700

Magnus Westerlund wrote:
> The problem with using ECN is that most of the sent 
> connectivity checks will be dropped by the external side of 
> NATs or hosts that don't have an open port or for which 
> the arrived packet is garbage and discarded.  Only 
> on packets where STUN requests are answered are there any 
> possibilities to use ECN to notice that there are some 
> issues on the path.

Right.  But if there are multiple paths which /do/ work, 
ECN could help provide additional information about which of
those paths is better.  ICE does allow each end to do some
optimization of its decisions, and communicate its decision
to the other end via the USE-CANDIDATE attribute.  Right
now about all the controlling agent could learn from the
network is the round-trip time of a Request/Response.  ECN
would improve that.

I could imagine that if ECN support were added to ICE, we 
would define another STUN attribute that is conveyed in 
the STUN Response.  This new attribute would communicate
the ECN bits of the received STUN Request back to the
controlling (ICE) agent.

-d