Subject: [Tsvwg] SCTP Multihoming issue

From: Janardhan Iyengar <[email-masked]>
Date: Tue, 21 Aug 2001 14:01:54 -0400 (EDT)

Hi all,

This a problem which I saw happen in simulations, and seems like an issue
that could come up everytime the primary on a multihomed SCTP endpoint is
switched.

SCTP peers A and B are multihomed, and assume that A is sending to B, on
destination IP0. At some point in time, the application decides to switch
the primary to destination IP1. At this point, 'A' could have packets
which have already been passed on to the network layer, and are buffered
now, waiting to be put on the line. The switching of the primary causes A
to load a new set of congestion control parameters, and most likely causes
A to go into slow start. 'A' can now send packets to destination IP1, and
these packets reach B faster than the ones sent to IP0, since those were
buffered and were being put on the line, when the new packets were sent to
IP1. B starts sending gap-acks to A, and this causes A to fast rtx packets
unnecessarily.

This problem is much more emphasized when a switch is done from a slower
link to a faster one. Ivan and I drew up a situation in which the
switching of the primary caused the bandwidth to change from 10Mbps to
100Mbps, with a propagation delay of 20ms, and there were as many as 8
packets fast rtx'd unnecessarily on the new link. Also, this prevents the
sender from sending new data, since the retransmissions take up cwnd
space.

regards,
jana

-----------------------------------------------------------------------
Janardhan R. Iyengar				   [email-masked]
University of Delaware		       http://www.cis.udel.edu/~iyengar
-----------------------------------------------------------------------

_______________________________________________
tsvwg mailing list
[email-masked]
http://www1.ietf.org/mailman/listinfo/tsvwg

---

From: [email-masked]
Date: Wed, 22 Aug 2001 03:10:54 +0300

Hi all there!

	We don't really know if this is something that we should take care
of or not, but in any case, Jana was making some simulations and he realized
about this "anomalous" behavior... To know if the effects could be
important, we were making some numbers about what we thought could be a
"standard" environment...

                      20 ms delay
     +--------+                       +--------+
     |        |-----------------------|IP0     |
     |        |       10 Mbps LAN     |        |
     | SCTP A |                       | SCTP B |
     |        |                       |        |
     |        |-----------------------|IP1     |
     +--------+      100 Mbps LAN     +--------+

	SCTP A is the data sender, and it is connected to SCTP B through two
links, a 10 Mbps and a 100 Mbps Ethernets. The delay of both paths (from
SCTP A to SCTP B and viceversa) is 20 ms. Everything will be simplified...
these are rough numbers... :-0

	Imagine that A's Primary Address is IP0, and it is sending DATA
chunks to that address when a change in the Primary Address happens, so now
IP1 is the primary address (or the uppers user simply specifies that IP1
must be used to send the next SSNs) and the next chunks go to that
destination. We could think as an example, that SCTP A is a HTTP server, and
for some reason is sending some file to IP0 and another file must be sent
using the other address, IP1... :-/

	As the speed of the 10 Mbps LAN is limited, SCTP A can be giving
packets to the 10 Mbps Ethernet card and it will take some time to transmit
then, so imagine that SCTP fragments a 100 KB file into pieces so that the
Ethernet frames are 1250 bytes long (to make calculation easier) and when
the change in the Primary address happens, there are 50 Ethernet frames
buffered and waiting to be transmitted, each containing a single DATA chunk
(TSNs 1-50)...

	At 10 Mbps each of such frames would take (1250*8)/(10E7) = 1 ms.
Let's call "time 0ms" at the time when the first of such frames is started
to be transmitted. Considering that cwnd of IP0 is, let's say 64KB, and rwnd
doesn't limit at all, those 50 frames will be transmitted between time 0ms
and time 50ms. They will arrive to SCTP B between time 20ms and time 70ms.

	Meanwhile, as IP1 hasn't been used, let's say that its cwnd is
enough to have two of those frames outstanding. TSNs 51 and 52 will be put
on the line from time 0ms to time 0.2ms. They arrive to SCTP B from time
20ms to time 20.2ms. Let's say the order of arrival of the first TSNs is: 1,
51, 52, 2, 3, 4, ...

	Forgetting about processing time, SCTP B sends back a SACK to SCTP A
for TSN 1 (the SACK for TSN 0 was being delayed) at time 21ms, we forget
about the time to put in the line a small SACK chunk, and it will reach SCTP
A at time 41ms. Meanwhile, when TSNs 51 and 52 arrive to SCTP B, they
originate a gap, and so two SACKs are created and sent back to SCTP A
through the fast line. They are created at time 20.1ms and 20.2ms and arrive
to SCTP A at time 40.1ms and 40.2ms (even before than the SACK for TSN 1).
They will increase the missing count for TSNs 1-50 (2 strikes). These SACKs
modify cwnd so now 4 frames fit, and TSNs 53-56 are sent to IP1 from time
40.1ms to time 40.5ms and will arrive to SCTP B from time 60.1ms to time
60.5ms...

	TSNs 2-50 keep arriving to SCTP B, one every millisecond, and all of
them report gaps, but using the modified Fast Retransmit (Implementors
Guide), they won't increase any Fast Retransmit counter :-). Those SACK will
arrive to SCTP A from time 42ms to time 90ms.

	The four TSNs sent to IP1, arrive from time 60.1ms to time 60.5ms as
said before, and the SACK for them are sent from time 60.2ms to time 60.5ms,
arriving to SCTP A from time 80.2ms to time 80.5ms. At that time, on the
other line, SACK up to TSN 40 (arrived at time 80ms) will have arrived, and
so, SACK for TSN 53 makes the third strike on TSNs 41-50. Cwnd grows up to 5
frames, there are only three outstanding frames so TSNs 57 and 58 are sent
in time 80.2ms-80.4ms. SACK for TSN 54 makes the forth strike on TSNs 41-50,
and so, all of them are marked for fast retransmission. The arrival of the
other SACKs increase cwnd of IP1 up to 8 frames, two are already used (TSNs
57 and 58), and so TSNs 41-46 are fast retransmitted from time 80.4ms to
time 81ms... :-(

	I'm seeing now that we did some mistakes in our calculations... but
let's continue. From time 81ms to time 86ms the SACK for TSNs 41-46 arrive,
so they are not outstanding any more on IP1, and they are replaced by TSNs
47-50, and TSNs 59-60, sent at times 81-81.1ms 82-82.1ms, 83-83.1ms,
84-84.1ms, 85-85.1ms and 86-86.1ms... But SACK for TSNs 47-50 arrive at time
87-90ms, and so TSNs 61-64 are sent at times 87-87.1ms, 88-88.1ms, 89-89.1ms
and 90-90.1ms...

	Summarizing, TSNs 41-50 (10 DATA chunks) are unnecessarily resent...
Also, as they are sent instead of new chunks, they delay somehow the
delivery of new data (TSN 64 is sent in time 90-90.1ms, while it could have
been sent in time 80.9-81ms)... :-<

	I'm pretty sure that I committed mistakes somewhere... Anyway, I
don't think that the situation is especially the worst case, and you are
resending 10 TSNs... It is worth noting that even if both links are 10 Mbps,
the result would be pretty similar (I think that 7 would be Fast
Retransmitted). Actually I think that in that case the rule is that a total
of (time of putting on the line the buffered TSNs - RTT - 3) TSNs are
sent... If you increase the frames buffered, or reduce the delay, this value
can be really big... 100 frames, 1ms a frame, 30ms delay -> 100*1 - 30*2 - 3
= 37 TSNs retransmitted... :-<

	Jana and me were thinking about a possible "kind of solution" to
avoid it... It has some weaknesses (especially when the last TSN sent to the
old primary address, in the example TSN 50, is really lost)... I will
explain it in a separate mail, so maybe some wise people there can think
about a real solution to this (which I think is not that easy)... :-0

	Thanks for your time! (I know it might be painful to read this whole
mail... :-(, trying to be clear I write too much sometimes...)

	I'm coming back in a while!

	BR Iván Arias Rodríguez

> -----Original Message-----
> From: ext Janardhan Iyengar [mailto:[email-masked]]
> Sent: 21. August 2001 21:02
> To: [email-masked]
> Subject: [Tsvwg] SCTP Multihoming issue
> 
> 
> Hi all,
>

---

From: Jon Berger <[email-masked]>
Date: Thu, 23 Aug 2001 13:30:06 +0100

I think that SCTP needs to solve the "network engineering" issues such as
stopping one protocol/host/association from using all of the available
bandwidth to the detriment of all other protocols/hosts/associations.  I
think that everyone is agreed that this is an issue for further study when
using two paths simultaneously.
Other than that, I think the rest of the discussion seems to be an
implementation issue, not a protocol issue.  Do whatever seems reasonable to
you/your applications as long as it doesn't break SCTP's rules.

Jon

-----Original Message-----
From: [email-masked]
[mailto:[email-masked]]
Sent: 23 August 2001 13:18
To: [email-masked]
Subject: RE: [Tsvwg] SCTP Multihoming issue

	Hi!

	I have what might be a possible solution to the problem of changing
addresses to gain throughput... Why not simply set the cwnd to 2*MTU when a
change in the primary is done? I mean, if you use another address than the
primary to send anything which is not a retransmission, then you set cwnd to
2*MTU... Or to some other low value... or you halve it, or whatever...

	This way you won't almost hurt at all "well-behaved" applications...
If you start using another address different than the primary to send some
_new_ data, it would mean that the primary is down. In that case you might
have done some retransmissions that could make the cwnd of the secondary
address grow a little... just a little... But as the RTO of the primary is
growing with every time out, surely when the last time out occurs (the one
that tells you that the primary address is down), you haven been RTO max (1
minute) waiting for the SACKs that didn't come, and the cwnd of the
secondary address will be for sure 2*MTU (if the secondary is working
normally, its RTO will be RTO min, 1 second, so you have halved it 60 times
while waiting...)

	This way, you keep the same behavior than now when SCTP is behaving
as it is expected, and I think you would avoid the problem... And the change
maybe takes one line of code...

	If the user tells SCTP to use another address then it is a little
bit tricky... Well... The first thought that comes to my mind now is that
maybe the solution to the whole thing is just not letting cwnd grow more
than, let's say up to 8*MTU if it is not the primary...

	What do you think?

	BR Iván Arias Rodríguez

> From: ext [email-masked] [mailto:[email-masked]]
> Sent: 23. August 2001 12:47
> To: [email-masked]
> Subject: Re: [Tsvwg] SCTP Multihoming issue
> 
> 
> Qiaobing Xie wrote:
> > 
> > Essentially we are talking about the traffic behavior when 
> _multiple_
> > destination addresses are involved _simultaneously_ in data 
> transmission
> > (this can happen either during the changeover of the 
> primary address or
> > when the upper layer intentionally directs data across different
> > destination addresses).
> > 
> > Throughout the development of SCTP, we made it very clear that this
> > "simultaneous, multi-path data transmission behavior" was a research
> > topic (I remember Chip Sharp suggested that this could be a good
> > research project for a Ph.D candidate).
> > 
> > There was no intention in SCTP design to address how traffic control
> > should be done under such condition. This is the primary reason we
> > removed round-robin from SCTP and made it a guiding 
> principal that "SCTP
> > multi-homing was NOT meant for traffic sharing, but only for path
> > redundancy!"
> > 
> > I don't think we know enough about "simultaneous, multi-path data
> > transmission behavior" over Internet, nor do I think the 
> time is mature
> > for us to talk about "solution" to modify SCTP for this purpose.
> > 
> > Therefore, my suggestion would be to stick with the 
> principal that "SCTP
> > multi-homing is only for path redundancy." If your primary 
> address fails
> > and you have to failover to an alternate, you shouldn't be 
> surprised to
> > see some disturbance in the traffic pattern.
> 
> I think I need to clarify the proposed solution.  Actually, the 
> solution is not for the observation Ivan and Jana saw.  It is for
> patching the weakness that congestion control in one association can
> be easily gotten around by switching primary addresses.  I agree that
> doing load sharing using multiple addresses is a research issue.  But
> I think that allowing an app to circumvent congestion control by
> switching primary addresses is a weakness.  Yes, an app can do the
> same thing by opening multiple associations, just like opening 
> multiple TCP connections.  But to have the weakness within a single
> association is a different issue.
> 
> If the rationale behind having multiple addresses is for redundancy,
> then it is reasonable to limit the number of data in the network
> when the switch happens.  Consider the case that the swtich of
> addresses happens because of real network outage.  Then it is 
> reasonable to slow start using a new primary address.  This is
> definitely a very conservative behavior.  If the path to the new 
> primary address is completely different from the path to the old
> one, I agree that we do not need to be that conservative.  But since
> SCTP cannot, in general, have this kind of info, it seems to me that
> it is better for us to be conservative at this point of SCTP
> deployment.
> 
> 							K. Poon.
> 							
> [email-masked]
>

---

From: [email-masked]
Date: Thu, 23 Aug 2001 15:41:28 +0300

Hi Jon,

	But the problem is that you don't have to break any SCTP rule to do
it... My implementation is "well behaved" but you know that congestion
control in Internet depends on the habits of all the users... Unfortunately
not everybody is willing to behave in the "right way" to achieve an overall
throughput gain, when he could achieve a bigger throughput in detriment of
the others...

	I think that limiting the cwnd of an address which is not the
primary matches the idea of using multihoming only as a backup...

	BR Iván Arias Rodríguez

> From: ext Jon Berger [mailto:[email-masked]]
> Sent: 23. August 2001 15:30
> To: Arias-Rodriguez Ivan (NRC/Helsinki); [email-masked]