Subject: RE: Kicking TCP

From: [email-masked] (Keith Scott)
Date: Tue, 07 Mar 2000 14:17:18 -0500

We in fact have something akin to this in the Space Communications Protocol 
Standard (SCPS, http://www.scps.org) protocol definition.  We haven't 
tested it on any large scale yet, but the idea was to allow the link layer, 
either through inter-layer signalling or by setting an entry in a MIB, to 
inform the transport layer about the status of an outbound link.  When a 
link goes down, the network layer might reroute, but the transport layer 
could also essentially suspend the retransmission timer and probe (akin to 
persist) to determine when the link becomes available again.  Using an LRU 
queue and something like an ICMP message would allow a router serving many 
flows to inform several (hopefully all) users of a link of that link's status.

 From the SCPS-TP Specification 
(http://www.ccsds.org/blue_books.html).  The reference [10] mentioned is 
the specification for the SCPS network layer.

3.4.3 LINK OUTAGES
When the sending TCP receives an indication of a link outage, either from 
inter-layer
signaling or from a MIB-query, the sending TCP
   a) shall suspend transmission;
   b) shall suspend the retransmission timer;
   c) may continue to queue user data for transmission, to the extent that 
retransmission
         buffer space permits (this is a local implementation option);
   d) shall periodically, at a locally determined rate (recommended to be 
some small
         multiple of the previous retransmission timeout), send a 
link-probe segment,
         consisting of the first octet of unacknowledged data:
       1) if no unacknowledged data exist, no link probe shall be sent;
       2) if data are submitted for transmission during a link outage, the 
sending of link
         probes shall resume;
   e) shall not interpret link probe segments as retransmissions for the 
purposes of
         modification of the retransmission timeout or for invocation of 
slow start or
         congestion avoidance;
   f) may resume normal communication when the sending TCP
       1) receives a 'link restored' or 'link redirect' control message 
(refer to reference [10]);
       2) determines that the link is available from a MIB-query (refer to 
5.2.2.8 of this document);
       3) receives an acknowledgment that arrives at least one RTT after 
the commencement
         of the link outage.

                 --keith

At 04:02 AM 03/07/2000 -0800, Marie-Jose Montpetit wrote:
>I would tend to agree that some simulations may be needed.
>
>However the kicking strategy is closely related to something I have been
>thinking for a while on how to "advertise" that a link is down or degraded.
>ICMP extensions were also our first idea or raising the "cost" of a route
>when the link is down (when multiple routes are available).
>
>Maybe what is needed in LINK is a section on suggested mechanisms to survive
>capacity degradation?
>
>Marie-Jose

---

From: Zachary Amsden <[email-masked]>
Date: Tue, 07 Mar 2000 11:56:06 -0800

There is a mechanism for sending redirects within the protocol framework 
itself - ICMP source quench can be used to indicate the loss of a flow, and 
ICMP redirect may be sent as an indicator that the flow has been restored.

-- 
Zachary Amsden  [email-masked]  3-6919  31-2-510  Core Protocols

---

From: [email-masked] (Keith Scott)
Date: Tue, 07 Mar 2000 15:07:54 -0500

Yes, but if you do this you have to take the hit from the source quench and 
go through slow-start again.  Using our method (the ICMP-style one, say), a 
source gets notified that a link it was using has gone dead and immediately 
suspends its timers and begins probing.  When the link comes up again, 
transmission resumes with no cut in cwnd.  A selective negative 
acknowledgement (SNACK) scheme helps us recover from packets in transit 
when the link went down.  Using SACK to complement SNACK would probably do 
even better.

                 --keith`

---

From: Ravi Prakash <[email-masked]>
Date: Tue, 7 Mar 2000 14:12:58 -0600 (CST)

Yes, I agree with this. We proposed a similar approach in our WCNC'99
paper (co-authored with Meghana Sahasrabudhe) for mobile ad hoc networks. 
There, we also observed that if a path is disrupted for an extended period
of time, the routing protocol will discover this fact and set the
distance metric to infinity. In ad hoc networks as each node also serves
as a router such an ICMP message can be generated locally. 

Of course, Phil's idea preceded our similar observation by several years.

Thanks,

----Ravi.

On Tue, 7 Mar 2000, Zachary Amsden

> ICMP redirect may be sent as an indicator that the flow has been restored.

========================================================================
Ravi Prakash ([email-masked])		www.utdallas.edu/~ravip
Department of Computer Science
The University of Texas at Dallas
Richardson, TX 75083-0688.  Phone: (972) 883-2289,   Fax: (972) 883-2349

---

From: Zachary Amsden <[email-masked]>
Date: Tue, 07 Mar 2000 12:41:33 -0800

> when the link went down.  Using SACK to complement SNACK would probably do 

how about using ICMP unreachable with a new code for Network Temporarily 
unavailable - if this is received, TCP should save it's current window, 
temporarily cut the window to zero, and begin sending ACK probes at a regular 
rate.  When it receives an ACK in return (see note below), it restores it's 
old window and resumes transmission.

*Note - you could send some type of message Network Available Again, but if 
the transmitting hosts resends in response to that, it may act too eagerly - 
it may take some time for routes to be renegotiated on the other end of the 
connection, causing the data to be misrouted and lost.  That will result in a 
slow start for TCP, which is what we were trying to avoid.  Getting an ACK in 
response to a probe ensures the network is stable again.

The real question is what to use for the probe timeout value - perhaps a 
function of the computed RTT for the connection?

---

From: Dan Grossman <[email-masked]>
Date: Tue, 07 Mar 2000 16:50:23 -0500

Seems to me that resuming without changing cwnd assumes that the congestion 
state hasn't changed during the outage.   That seems unlikely.  Wouldn't it be 
better to reduce cwnd to ssthresh?

---

From: Phil Karn <[email-masked]>
Date: Tue, 7 Mar 2000 23:28:56 -0800

>ICMP redirect may be sent as an indicator that the flow has been restored.

I don't think this is the intended use of ICMP redirect. According to
RFC1122, a redirect will modify a routing table, but it is not one of
the ICMP messages passed upstairs to the transport layer. Are you
proposing that it taken on a new meaning?

I'm not necessarily opposed to such a proposal, but the LINK document
is supposed to concentrate on telling subnet designers what they
should do, complementing the other work in PILC on possible changes to
the Internet protocols to better support unconventional subnetwork layers.
Perhaps your suggestion should be directed to one of the other documents?

Phil

---

From: Phil Karn <[email-masked]>
Date: Tue, 7 Mar 2000 23:36:13 -0800

>old window and resumes transmission.

I agree with Dan Grossman; I'm not sure you want to save and restore
the old TCP state, as opposed to letting it restart slow-start.

During the outage state, there is probably little harm in letting the
sending TCP probe the link with normal retransmissions. Because each
retransmission will be backed off exponentially, the load will be
limited even if full packets are used in the probes.

Phil

---

From: [email-masked] (Keith Scott)
Date: Wed, 08 Mar 2000 08:35:01 -0500

Responding to multiple messages:

Zachary: Your new ICMP message type is very similar to the "Link Outage" in 
the SCPS draft, but using a received ACK instead of a "Link Restored" 
ICMP/MIB Query to notify TCP that the link is back?  OK.  Although if you 
simply 'resotre your old window and resume transmission' you'll have the 
same problem of being too aggressive, no?

The points about being too aggressive on a "Link Restored" are quite 
valid.  In some special cases it's justified (e.g. the link that goes out 
does so for a short period of time then comes back up; routes don't change 
and there aren't too many other flows/enough time that they can eat up the 
bandwidth from suspended flows).  For specific environments like some 
spacecraft comms, this is valid.  Could also be valid for some last-hop 
wireless scenarios (Phil?).

If the link is out for too long, there are a lot of flows sharing routers 
used by suspended flows, etc., then resuming at full speed is probably too 
aggressive (and is bursty regardless), and you would definitely want to 
reduce cwnd or take some other intelligent action.

                 --keith

---

From: Vincent Magret <[email-masked]>
Date: Wed, 08 Mar 2000 10:45:31 -0600

Why sending a probe when the information could be sent for the node in the
network that monitors the broken link. Just put the TCP sender connection in idle
state for a certain time, after which the connection will be broke down. The
question is then How long?

Vincent

---

From: Zachary Amsden <[email-masked]>
Date: Wed, 08 Mar 2000 10:41:34 -0800

> the old TCP state, as opposed to letting it restart slow-start.

That was the intention of my original idea - to reduce cwnd and send periodic 
(rather than backed-off) probes, and do a slow-restart upon receipt of an ACK.

> limited even if full packets are used in the probes.

During the outage state, if the router is queuing the data as suggested, upon 
restoration, any queued data will be sent as a burst to the other side, and 
the resulting ACKs will cause the window to re-inflate.  So if you have a 
buffering router, no protocol modifications are really necessary to get you 
going again.  So yes, restoring cwnd to the pre-outage value with a buffering 
router would cause the window to possibly double what the window was before 
congestion, which would be a bad thing.

With no buffering in the router, saving/restoring cwnd via link down/up 
notifications seems like a good idea.  At most only hosts that recognize the 
notification scheme will start retransmitting with prior windows, so hosts 
that don't will just backoff and slow-start.  So there shouldn't be any 
problem with congestion after link up notification if there wasn't any 
congestion when the link went down.

The notification scheme is really just an implementation detail, and I'm just 
wondering whether it can be done within the current framework by being 
creative without breaking too many semantics.

---

From: Phil Karn <[email-masked]>
Date: Wed, 8 Mar 2000 11:28:30 -0800

>the resulting ACKs will cause the window to re-inflate.  So if you have a 

Only if all the data is queued during the outages. I propose having the subnet
queue only the last packet on each flow, or alternatively some small number
of packets N, with the older packets discarded as new ones arrive.

Phil