Subject: RE: [Tsvwg] SCTP Multihoming issue

From: [email-masked]
Date: Thu, 23 Aug 2001 15:18:04 +0300

Hi!

	I have what might be a possible solution to the problem of changing
addresses to gain throughput... Why not simply set the cwnd to 2*MTU when a
change in the primary is done? I mean, if you use another address than the
primary to send anything which is not a retransmission, then you set cwnd to
2*MTU... Or to some other low value... or you halve it, or whatever...

	This way you won't almost hurt at all "well-behaved" applications...
If you start using another address different than the primary to send some
_new_ data, it would mean that the primary is down. In that case you might
have done some retransmissions that could make the cwnd of the secondary
address grow a little... just a little... But as the RTO of the primary is
growing with every time out, surely when the last time out occurs (the one
that tells you that the primary address is down), you haven been RTO max (1
minute) waiting for the SACKs that didn't come, and the cwnd of the
secondary address will be for sure 2*MTU (if the secondary is working
normally, its RTO will be RTO min, 1 second, so you have halved it 60 times
while waiting...)

	This way, you keep the same behavior than now when SCTP is behaving
as it is expected, and I think you would avoid the problem... And the change
maybe takes one line of code...

	If the user tells SCTP to use another address then it is a little
bit tricky... Well... The first thought that comes to my mind now is that
maybe the solution to the whole thing is just not letting cwnd grow more
than, let's say up to 8*MTU if it is not the primary...

	What do you think?

	BR Iván Arias Rodríguez

> -----Original Message-----
> From: ext [email-masked] [mailto:[email-masked]]
> Sent: 23. August 2001 12:47
> To: [email-masked]
> Subject: Re: [Tsvwg] SCTP Multihoming issue
> 
> 
> Qiaobing Xie wrote:
> > 
> > Essentially we are talking about the traffic behavior when 
> _multiple_
> > destination addresses are involved _simultaneously_ in data 
> transmission
> > (this can happen either during the changeover of the 
> primary address or
> > when the upper layer intentionally directs data across different
> > destination addresses).
> > 
> > Throughout the development of SCTP, we made it very clear that this
> > "simultaneous, multi-path data transmission behavior" was a research
> > topic (I remember Chip Sharp suggested that this could be a good
> > research project for a Ph.D candidate).
> > 
> > There was no intention in SCTP design to address how traffic control
> > should be done under such condition. This is the primary reason we
> > removed round-robin from SCTP and made it a guiding 
> principal that "SCTP
> > multi-homing was NOT meant for traffic sharing, but only for path
> > redundancy!"
> > 
> > I don't think we know enough about "simultaneous, multi-path data
> > transmission behavior" over Internet, nor do I think the 
> time is mature
> > for us to talk about "solution" to modify SCTP for this purpose.
> > 
> > Therefore, my suggestion would be to stick with the 
> principal that "SCTP
> > multi-homing is only for path redundancy." If your primary 
> address fails
> > and you have to failover to an alternate, you shouldn't be 
> surprised to
> > see some disturbance in the traffic pattern.
> 
> I think I need to clarify the proposed solution.  Actually, the 
> solution is not for the observation Ivan and Jana saw.  It is for
> patching the weakness that congestion control in one association can
> be easily gotten around by switching primary addresses.  I agree that
> doing load sharing using multiple addresses is a research issue.  But
> I think that allowing an app to circumvent congestion control by
> switching primary addresses is a weakness.  Yes, an app can do the
> same thing by opening multiple associations, just like opening 
> multiple TCP connections.  But to have the weakness within a single
> association is a different issue.
> 
> If the rationale behind having multiple addresses is for redundancy,
> then it is reasonable to limit the number of data in the network
> when the switch happens.  Consider the case that the swtich of
> addresses happens because of real network outage.  Then it is 
> reasonable to slow start using a new primary address.  This is
> definitely a very conservative behavior.  If the path to the new 
> primary address is completely different from the path to the old
> one, I agree that we do not need to be that conservative.  But since
> SCTP cannot, in general, have this kind of info, it seems to me that
> it is better for us to be conservative at this point of SCTP
> deployment.
> 
> 							K. Poon.
> 							
> [email-masked]
> 
> _______________________________________________
> tsvwg mailing list
> [email-masked]
> http://www1.ietf.org/mailman/listinfo/tsvwg
> 

_______________________________________________
tsvwg mailing list
[email-masked]
http://www1.ietf.org/mailman/listinfo/tsvwg

---

From: Randall Stewart <[email-masked]>
Date: Thu, 23 Aug 2001 08:54:58 -0500

Ivan:

First of all I don't think your idea will suceed. In the
normal case your cwnd is going to be 2*MTU on the backup
destinations. Imagine a endpoint with 10 addresses on
each side. Now I send 2 MTU's to one, change primary... etc.
And of course limiting things to 8 * MTU does not help either...

You have not prevented anything since now I can have 20 * MTU
outstanding.

I think Kacheongs idea (with my little modification) is more the
right way to go. If there is anything to be done here at all...
I am still not sure :)

R

-- 
Randall R. Stewart
[email-masked] 815-342-5222 (cell phone)

---

From: Qiaobing Xie <[email-masked]>
Date: Thu, 23 Aug 2001 13:59:55 -0500

Randy,

The issue of someone abusing SCTP multi-homing to gain unfair share of
bandwidth is definitely something more worthwhile addressing than that
of minimizing unnecessary retransmission during primary changeover :-)

More comments below...

-Qiaobing

> right way to go. If there is anything to be done here at all...

But K's idea does not prevent anything either. The sender does NOT need
to perform a switch_primary to do multi-path data transmission. He
simply specifies a different destination address every time he sends.
For example, he can cheat for bandwidth by simply round-robin among all
the peer destination addresses without invoking a switch_primary. How
will K's idea help in this case?

---

From: Randall Stewart <[email-masked]>
Date: Thu, 23 Aug 2001 14:44:29 -0500

Comments below..

> will K's idea help in this case?

I guess I thought about this part but did not write it down :<

Basically I would consider a override destination type send (what
you question above) to be the same as a switch primary. Anytime a
new send shows up with the flag set (if the implementation supports
this override flag that is) then treat it just as if they had done a
switch
primary. The same rules can be used.. i.e. you keep track of where the
cum-ack was in the new destination and what extra credit to add to
the "caculated flight size"... i.e. the actual flightsize on the
network plus the credited flight size of the primary...

It will definetly make more bookkeeping when handling SACK's... el
yucko.. but it would prevent both types of abuses..

But is this an implementation issue or one for the spec. Maybe
Kacheong is right... without it being in the spec you end up with
Vendor-X claiming they are conformant and they get X% better bandwidth
on multi-homed endpoints.

Of course even if you change the spec you still can cheat but that
would then be a violation of the spec... maybe this is something
to add when SCTP goes draft standard...

R

---

From: Qiaobing Xie <[email-masked]>
Date: Thu, 23 Aug 2001 15:21:00 -0500

To avoid all this trouble, why not simply ban the data sender from any
manipulation of the primary address.  This means that only the data
receiver can switch primary through the Set_Primary message (part of
Add-ip extension). This makes sense because after all the receiver owns
the destination addresses and in most cases is the one that would want
to re-direct inbound traffic to a different local address.

I think this can effectively prevent the abuse without adding any
complexity to the protocol or implementation.

-Qiaobing

Randall Stewart wrote:
...

---

From: "Lars Eggert" <[email-masked]>
Date: Fri, 24 Aug 2001 09:03:49 -0800

> of minimizing unnecessary retransmission during primary changeover :-)

I've just started following this mailing list and SCTP in general, so
please excuse my ignorance: Why was in-protocol support for multihoming
deemed necessary for SCTP in the first place? (If it wasn't there, this
problem wouldn't exist.)

I'm asking since TCP and UDP seem to get by fine without addressing this.
Multihoming seems like a network layer thing to me, so maybe it should be
handled there?

Thanks,
Lars
--
Lars Eggert <[email-masked]>               Information Sciences Institute
http://www.isi.edu/larse/              University of Southern California

---

From: Qiaobing Xie <[email-masked]>
Date: Fri, 24 Aug 2001 11:35:11 -0500

Hi, Lars,

In-protocol multi-homing support is one of the primary reasons for
developing SCTP. The purpose is to gain path-level fault tolerance. This
idea originated from SIGTRAN working group, where people were looking
for a reliable way to carry SS7 signaling message across IP network.
Both TCP and UDP were examined carefully but were found not to have
adequate support for path-level redundancy which SS7 traffic requires
(for instance, you need to have your public telephone system survive a
NIC card failure in one of your critical signaling path).

If you are interested in more details of those analysis and discussion,
you may want to dig into the SIGTRAN mail archive (not tvswg archive).

-Qiaobing

---

From: Kacheong Poon <[email-masked]>
Date: Fri, 24 Aug 2001 10:18:58 -0700 (PDT)

> will K's idea help in this case?

My original idea is slightly different from what Randy sent out.  It
is more drastic, and this is the reason why I didn't send it to the list.
I was thinking to have one cwnd to cover all addresses.  This idea came
from the fact that the primary design goal for multihoming is for
redundancy.  This means that transmission normally will only happen between
primary addresses.  Having the cwnd covers all addresses will not affect
normal transmission.  In case when a switch needs to happen, it will slow
down transmission a little bit.  But I think it is fine.  More over, I
also prefer to have cwnd reset at 2 MTUs whenever a different address
is used to send.  Since there is only one cwnd, it also covers the case 
Qiaobing mentioned above, even if cwnd is not reset when a switch happens.

The reason I don't want to disallow an app to switch addresses is that
there are legitimate reasons to do so.  For example, an app checks the
status of the primary address once in a while (assuming a persistent
association) and see what it is.  The reason is that it is more expensive
to use the backup network path.  So if possible, it wants to switch back to
the primary path.  But it still relies on SCTP to do the automatic switch
over and path outage detection.

The idea is preliminary and drastic, so I didn't send it to the list.  I 
wanted to bounce it around first.  Anyway, what do people think about it?

							K. Poon.
							[email-masked]

---

From: Jacob Heitz <[email-masked]>
Date: Fri, 24 Aug 2001 10:25:49 -0700

Routers can provide path-level fault tolerance. If a
network element fails the routers can reconverge.

The real point is speed. For some failures, it will
take the routers a hello interval before they even know
there is a problem. This is much too long for signaling
transport. SCTP can send to an alternate address
immediately after just a single packet is dropped.

-- 
// Jacob Heitz Tel:510-747-2917 Ext:72917 Home:510-888-9429

---

From: Randall Stewart <[email-masked]>
Date: Fri, 24 Aug 2001 12:50:51 -0500

> network element fails the routers can reconverge.
I would agree routers can converge.. but even beyond this
Jacob... a router can NOT help you if the NIC card
of one of the machines fails... I think this
was Qiaobing's point...

R

---

From: Randall Stewart <[email-masked]>
Date: Fri, 24 Aug 2001 12:54:01 -0500

Kacheong:

Sorry if I sent this out to quick for you... I am against
having a single cwnd. This is to drastic in my mind. As I
mentioned with my proposed modification I may be more favorable...

Overall I am still not convinced this is a major problem... if
something needs to be done I would rather see a softer change...

R

---

From: "Lars Eggert" <[email-masked]>
Date: Fri, 24 Aug 2001 10:56:00 -0800

Qiaobing & Jacob,

thanks for the background information on this!

> NIC card failure in one of your critical signaling path).

But local hardware failures are all this seems to protect you against.
Even if you switch interfaces, you can't know that the packets will follow
separate paths past the first hop. Even to the first hop, packets may go
over the same bundle (or even fiber), if ISPs buy subleased capacity.

I'll start digging in the SIGTRAN archive, this must have been brought up
before.

Lars

---

From: Qiaobing Xie <[email-masked]>
Date: Fri, 24 Aug 2001 13:13:56 -0500

Kacheong Poon wrote:
....
> normal transmission.  

I think the heart and soul of the TCP-like (i.e., AIMD) congestion
control algorithm (accumulated from the research and Internet
experiences from past 3 decades or so) is the way how the cwnd is
dynamically probed and changed. However, most of those research and
experience came from single-path data transmission across Internet. I
guess the question is whether the same AIMD congestion control will work
well across multiple (potentially asymmetric) paths.

For example, let's assume the peer has two addresses IP0 and IP1 and the
data sender (local endpoint) uses a global cwnd and start the session
with IP0 as primary.

After the slow start phase and some data transfer, the cwnd converges
to, say, ~20*MTU of IP0. Now the data sender switches its primary to
IP1. According to your proposal, the data sender will start transmitting
data to IP1 with cwnd=20*MTU?! This is wrong, because IP1 may lead to a
totally different path and the intrinsic cwnd of this new path is
completely unknown at this point. For this exact reason, as Lixia
pointed out during the SCTP development, **the sender must go through
slow start phase again and re-probe the cwnd of IP1**. She was the one
who raised this issue and suggested to have a cwnd per destination
address (and I still totally agree with her). Before that, we did have a
global cwnd :-)

> In case when a switch needs to happen, it will slow

The same reason (i.e., user probing the reachability of a specific dest
address) has been given before. However, I think the user-triggered,
dest address-specific heartbeat serves the purpose adequately and is
much better than sending actual user data to probe the reachability of
an alternate address.

-Qiaobing

---

From: Joe Touch <[email-masked]>
Date: Sat, 25 Aug 2001 00:31:28 -0700

> But local hardware failures are all this seems to protect you against.

Further, as Lars mentioned, we've implemented solutions to this that
are robust on interface failures. The technique involves moving the
lasthop 'router' "inside" the end-host - having it perform routing
as well. The applications attach to a virtual interface inside the end-host;
the multiple NICs on the host are treated as if on this pseudo-router.

The configuration for such a system was discussed in an ICNP 1997 paper:
    "Dynamic Host Routing for Production Use of Developmental Networks"
    J. Touch and T. Faber, Proc. ICNP '97, Atlanta, Oct. 1997, pp. 285-292. 
    http://www.isi.edu/touch/pubs/icnp97/icnp97.ps

(PS - it was designed for overcoming NIC failures, in particular, in the
production use of experimental equipment)

Granted, as you mentioned, neither TCP nor UDP have sufficient support
for end-address redundency. Why expect that from the transport protocol, 
though?

Joe

---

From: Randall Stewart <[email-masked]>
Date: Sat, 25 Aug 2001 06:08:44 -0500

Qiaobing:

I think that Kacheong is also saying that when you did
a switch-primary you would reset cwnd back to 2 MTU...

I have thought on this a bit and have the following opinion:

1) It is the wrong approach. Being way to drastic since
   an easy way to escape this is to do excactly what
   current TCP implementations do that have multiple nic's
   on each end. Set up two (or more ) connections. When I
   do this I get the same effect as the current SCTP and
   switching primary's.

2) It would be easier to specify a limitation to the 
   "set.primary" command. Something along the lines
   that the set.primary MUST NOT take effect until
   the flightsize on the old primary drops to 0.

3) I am not really sure that this is a big issue. Since
   folks are still allowed to do this with TCP or even 
   SCTP using (1). Yes the application needs to be 
   setup special to deal with multiple connections/associations
   but this is also true for this switching primary's. In order
   to do all of this one would have to have code in the app that
   was a) aware of all the addresses of the peer b) kept track
   of what it had sent and switches or switches after every send.
   In fact this seems a harder problem then just setting up
   a pair of SCTP associations and just setting the primary
   different on each one...

The more I think on it I really don't see it has a problem...

R

---

From: Jacob Heitz <[email-masked]>
Date: Sat, 25 Aug 2001 09:54:18 -0700

There is another behaviour I would like to see permitted.
I think it is permitted, but I would like to get confirmation.

A sending SCTP is transmitting to the primary address.
The RTT expires and it retransmits to an alternate address.
Let's assume an alternate is available.
So far so good. Now, I want it to transmit ONLY heartbeats
to the primary address and to transmit all further packets
an alternate address. Only once a heartbeat
succeeds on the primary, will SCTP resume sending data to
the primary address.

I have an SCTP that insists on sending all first transmissions
to the primary address and only retransmissions to the
alternate addresses. This SCTP behaves unacceptably slowly
when the primary address fails until it finally declares
the primary address to be down.

I need multihoming to give me an immediate seamless recovery
in the case of failure of the primary address.

-- 
// Jacob Heitz. Tel:510-747-2917

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Sat, 25 Aug 2001 14:00:10 -0500

My strict interpretation of the RFC yeilds the same poor
behavior.  I also believe that other behaviors are permitted
and would be interested in hearing that confirmed.

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Sat, 25 Aug 2001 14:24:06 -0500

Randall,

Sliding window congestion control (cwnd) is always positioned at the
entry point to the network for TCP.  SCTP's problems with cwnd arrise
from associating cwnd with a destination address rather than an entry
point.

In implementation it seems a rather easy thing to do to associate cwnd
with an interface rather than associating it with a destination address.
If this is done, SCTP would be completely fair to TCP (which also has a
per interface cwnd).  Trying to get a per-destination cwnd to be fair to
TCP depends strongly on the network configuration, and is fair in some
configurations and not fair in others.

These problems were raised during the design of SCTP, but never
adequately addressed.

--brian

-- 
Brian F. G. Bidulock    ¦ The reasonable man adapts himself to the ¦
[email-masked]    ¦ world; the unreasonable one persists in  ¦
http://www.openss7.org/ ¦ trying  to adapt the  world  to himself. ¦
                        ¦ Therefore  all  progress  depends on the ¦
                        ¦ unreasonable man. -- George Bernard Shaw ¦

---

From: Randall Stewart <[email-masked]>
Date: Sun, 26 Aug 2001 17:34:47 -0500

Jacob:

Currently I don't see that there is any restriction
against setting up a "2 level" threshold. 

Theshold A and B.

The minute you need to do a retran, you have hit
A. You then engage your behavior you illustrate.
If you miss N more heartbeats (since you are not
sending data to the primary anymore) you then hit
threshold B i.e. the destination is unreachable.

This is a reasonable splitting of the current single
stage threshold and I see nothing wrong with it..

R

---

From: Kacheong Poon <[email-masked]>
Date: Mon, 27 Aug 2001 18:40:03 -0700 (PDT)

> well across multiple (potentially asymmetric) paths.

As I mentioned before, SCTP has no knowledge about the path a packet
traverses over the Internet.  The fact that it uses a different destination
address does not automatically mean that the paths are different.  Yes,
in an engineered network, the paths will be different.  But put SCTP in
the real world, then we are not sure.  That's why I suggested the
conservative approach.  Treat the peer addresses as a whole and use one
cwnd.  If a different address is used because of failure, cwnd starts
from scratch.

I guess that the fact that those paths partially overlap still provides
some redundancy.  Only when the overlapped path has problems, will SCTP's
mechanism fail to work.

> address (and I still totally agree with her). Before that, we did have a

Another reason why I sent the mail to Randy first.  I want to know
what the original idea of having the current design.  But note that the
proposed idea satisfies Lixia's concern.  I don't know the real original
global cwnd idea so I cannot comment on its problem.

> much better than sending actual user data to probe the reachability of

Actually, the reason I stated has nothing to do with user probing
the network.  It does not do any probing at all.  It just checks the 
current primary address and switches it back to the preferred address if
it decides to do so.

							K. Poon.
							[email-masked]

---

From: Kacheong Poon <[email-masked]>
Date: Mon, 27 Aug 2001 19:00:42 -0700 (PDT)

>    do this I get the same effect as the current SCTP and

As I mentioned before, they are 2 separate issues.  One is outside
the protocol (opening multiple connections/associations), the other
is within the protocol (sending data using different addresses within
an association).  IMHO, they should be discussed separately.  The
fact that they have similar effect does not mean that this is not
an issue of SCTP.

>    the flightsize on the old primary drops to 0.

As Qiaobing has mentioned, an app can just specify the destination
address to use.

>    a pair of SCTP associations and just setting the primary

Actually, it is much easier than above.  Think about a very simple
example.  An app opens two streams and the peer has 2 addresses. 
The app maps each stream to an address.  Then in this case, it is
equivalent to opening two TCP connections to the peer within one
association as both addresses have independent cwnd.  So now the 
question is whether this is the intention of the original design 
of SCTP's congestion control to allow this to happen.  Doing this, 
an app can send at twice the rate of just using different addresses 
as backup for failure given that the receiver's window allows that.

							K. Poon.
							[email-masked]

---

From: Kacheong Poon <[email-masked]>
Date: Mon, 27 Aug 2001 19:27:15 -0700 (PDT)

> for end-address redundency. Why expect that from the transport protocol, 

This is an interesting question.  I always find that multihoming
is a difficult issue (well, others may find it trivial...)  It seems
to me that the question is similar to asking why a transport protocol
needs to know the IP address a packet is sent to.  Can it just specify
the peer (using hostname?) to send to and let IP decide which address to
use?  At a higher level, why does an app need to know the IP address
to use to setup a connection?  For example, if the peer has multiple 
addresses, an app needs to pick one for TCP.  Is this just an 
implementation artifact?  Or is it intrinsic to Internet protcols?  If
it is intrinsic, is it possible to provide redundancy without a transport
protocol knowing about IP addresses, assuming that different addresses 
really lead to completely different paths?

							K. Poon.
							[email-masked]

---

From: Randall Stewart <[email-masked]>
Date: Tue, 28 Aug 2001 06:48:24 -0500

> fact that they have similar effect does not mean that this is not

No, I think they are the same issue. The use of two connections
in TCP is symantically the same thing as using multi-homing.
One you can do nothing about.. the other we have some small control
of...

> As Qiaobing has mentioned, an app can just specify the destination

I thought I had addressed this. You treat the "override.destination" the
same as a set.primary. In other words, if someone overrides the
destination then the previous primary's flight must drop to 0. Then
when the new data is queued to the new destination it becomes
the new primary..

You end up treatting the

send(....,override to A)

the same as

setsocket(new-primary A)
send(...)

This would in effect takee your sceario of "extra bandwidth theft"
completely
off the table but would still allow an implementation to fully use
bandwidth for redundancy purposes... Much less drastic and at the
same time we still accomplish fully what we wanted to with the
multi-homing...

> as backup for failure given that the receiver's window allows that.

My point here is that no matter how they do it, i.e. when I send
to a stream they must:

1) become aware of the various addresses
2) do some sort of mapping

and if they really are attempting to steal extra bandwidth
3) distribute the load over there mappings..

It IS extra work and if I setup 2 associations/connections it
is much easier.. I do a setprimary once  and then just
use the connections/assocaitions..

Your idea is way to drastic... I think mine provides a more
optimal solution that avoids not being able to fully use
the multi-homed endpoints during retransmissions...

R

---

From: Qiaobing Xie <[email-masked]>
Date: Tue, 28 Aug 2001 10:46:51 -0500

Kacheong Poon wrote:
.....
> That's why I suggested the

But remember that in RFC2960 the alternate address(es) is also used for
all retransmission (including fast retransmission). Are you suggesting
to set the global cwnd back to 2*MTU as well whenever retransmitting a
packet to an alternate address? If the answer is yes, I think the global
cwnd will collapse too often and hurt the overall throughput. If the
answer is no, then you may flood the alternate path if it is a far
thinner pipe than the primary path.

-Qiaobing

---

From: Qiaobing Xie <[email-masked]>
Date: Tue, 28 Aug 2001 11:07:28 -0500

Randall Stewart wrote:
.....
> when the new data is queued to the new destination it becomes

Still somewhat unclear to me. For example, if you have IP0 (primary),
and IP1 and your flightsize on IP0 is currently 20*MTU when the user
does:

 send(msg-1, ....,override to IP1)

by your suggestion, we would hold up msg-1 and wait for the flightsize
on IP0 to drop back to 0.

But, what if the user does the following:

 send(msg-1, ....,override to IP1)
 send(msg-2, ....)
 send(msg-3, ....)
 send(msg-4, ....)
 ....

Are we going to hold up msg-2, msg-3, msg-4 as well (they are sent to
the primary and possibly to different streams)? If yes, we would get
head-of-line blocking problem. If not, then there is a chance that the
flightsize on IP0 will never drop back to 0 and msg-1 will never be sent
out.

-Qiaobing

---

From: Randall Stewart <[email-masked]>
Date: Tue, 28 Aug 2001 12:10:56 -0500

> flightsize on IP0 will never drop back to 0 and msg-1 will never be sent

The answer is of course "yes"... msg2-4 would need to be held..

This is NOT a head of line blocking problem... at least not
in my mind.. since the user is doing something specifically
to attempt to get around cwnd ... i.e. being someone who
wants to take up more bandwidth then its entitled to.

And of course the counter argument is that .. the user wants
to do some sort of probe.. but we already have a ON-DEMAND
heartbeat that can be done on the alternate address...

So if the app is attempting to cheat I see no problem whatso ever
in giving a sever penalty to it. I can even see them doing

send(msg1, Override IP-1)
send(msg2, Override IP-2)
send(msg3, Override IP-3)
etc.

This would in effect create a chain of msg's and end up
forcing something very similar to Kacheong's single cwnd
proposal... If you think on this you end up with the same
situation since once you change addresses the cwnd would
plummet and if you have 20 MTU out, you will end up waiting
a awful long time to get your subsequent msg's out.

The nice thing about this is NO head-of-line blocking is
imposed if you use the interface properly.. i.e.

send(msg1, to-primary)
send(msg2, to-primary)

I am not even sure (anymore) that there is a reason to have
an override-and-send-to-IPx option. 

If you have it, it seems to me that there is nothing wrong
with imposing a penalty on the sender that when you
a) override or b) change primary's you incur a penalty... yes
you may even get H-O-L blocking.. but you don't have to.. it
was your choice to override things.

I can't think of any case that this is to bad... if the app
knows something is very wrong with an address (say via some
alternate means) then having the penalty for switching addresses
is not that big of deal. 

If the app needs to probe for reachability a user msg is not
necessary.. instead use the facility in the protocol demand-hb

R

---

From: Kacheong Poon <[email-masked]>
Date: Wed, 29 Aug 2001 15:14:34 -0700 (PDT)

> thinner pipe than the primary path.

I think we are getting into something pretty interesting here...  First
of all, if the retransmission is due to a timeout, cwnd is reset anyway.
It does not matter if it is a single cwnd or cwnd per destination address.
Note that this is the conservative (proper at this point of time I think) 
behavior as SCTP does not really know if there is partial overlap of paths
to those alternate addresses and where the congestion (the usual assumption
of drops) is.  Actually, even if the alternate cwnd is not reset, it is
probably at its lowest possible value as it is not used for data transfer
previously and cwnd should be reset every RTO if it is not used (assuming
following RFC 2581 instead of RFC 2960 which is quite aggressive...).

So your question is if SCTP uses an alternate address to do fast
retransmission, what is the behavior?  But I guess we should first answer
the question of when SCTP should use an alternate address to do fast
rertransmissions.

For SCTP to know about the congestion state of the network, it needs to
keep on sending data.  If fast retransmissions are sent to an alternate
address, it means that data sent to the primary address will be pretty
much emptied out from the network (this really depends on the drop pattern
and if new data can be sent) very quickly.  This means that the cwnd may
need to be reset because it is not used.  But if fast retransmissions
are still sent to the primary address, cwnd will be kept current.  And
as stated above, the cwnd of alternate address should probably start from
the lowest value.  Otherwise, if SCTP wants to fast retransmit several
packets, they will all go out at the same time, which is properly not
a good idea.  I believe using the alternate address is probably not faster
than sending to the primary address, actually it probably should not to
be network friendly.  Of course it assumes that fast retransmitted packets
are getting thru using the primary address.  This leads to following point...

Another minor point (which is quite weak (-:)...  If the path to 
primary address is really congested such that fast retransmitted packets 
are also dropped, it will induce a timeout quickly.  And as pointed out by
Jacob and Brian, it is better to use the alternate address after a timeout.
This makes SCTP switch address quicker.  If fast retransmitted packets are
sent to an alternate address, SCTP will still use the primary address
for new data transmission, albeit at a lower rate.  If the path is very bad,
it will have to switch anyway.  I guess it is better to switch sooner.  Well,
this all depends on the transfer pattern and state of the network.  

I believe switching address is another topic which is not easy to answer.  
But IMHO, automatic use of alternate address for retransmission is probably
not a good idea.  So I don't think using a single cwnd and reseting it
whenever a new address is used to send data is as problematic as it
seems to.  I always like simple thing, and using a single cwnd at this point
seems very simple to me (-:  When people come up with concrete ideas about
how to make better use of multiple addresses (not just for retransmission
backup), the single cwnd can be removed to accommodate that.  But there
is no such idea at this point.

							K. Poon.
							[email-masked]

---

From: Kacheong Poon <[email-masked]>
Date: Wed, 29 Aug 2001 15:26:03 -0700 (PDT)

> One you can do nothing about.. the other we have some small control

Another way to phrase this is, one is outside the protocol, the
other is inside the protocol.  IETF controls protocol design.  Well,
it seems that we need to do something about it...

> when the new data is queued to the new destination it becomes

This seems to be to be a change in the semantics as mentioned in
RFC 2960.  And this definitely requires a change in the wording in
the socket API draft.  

> the multi-homed endpoints during retransmissions...

Before we actually try to nail down a solution, we should first 
decide if this is a problem.  Randy's previous mail suggested
that this is not a problem.  How about other people's opinion?

If we decide that this is indeed a problem which we should address
in the protocol level, then we need to work on the root cause.

Qiaobing mentioned that we should just get rid of the feature which
allows app to specify which address to use to send to the peer.  I 
suggested a minor point that switching the primary address should still
be allowed.  And when an app switches the primary, the cwnd should
be reset.  This seems to be very simple, and just requires a change
in the SCTP API.  And this does not prevent future use of multiple
addresses for load balancing, as long as it is done within SCTP.
This means that SCTP will have complete control on what destination
address to use to send to peer.  An app can only use setting primary
peer address to make a hint to SCTP for the decision.

Alternatively, Randy proposed an implementation patch and I proposed
a very conservative approach in the use of multiple addresses.  So what
do other people think?  Any other suggestion?  Or comment on the
problem?

							K. Poon.
							[email-masked]

---

From: Jacob Heitz <[email-masked]>
Date: Wed, 29 Aug 2001 19:52:26 -0700

You make a very interesting point about retransmission
due to timeout vs fast retransmission.

Timeout retransmission is often due to an outage or
congestion. I support retransmission to an alternate
address.

Fast retransmission is due to a dropped packet, but
the path is still very much alive. I support retransmission
to the same address in this case.

-- 
// Jacob Heitz, Planet of the APX. Tel:510-747-2917