Subject: Re: [Tsvwg] SCTP applicability statement

From: Phillip Conrad <[email-masked]>
Date: Mon, 05 Nov 2001 11:00:11 -0500

A bit of comparison between TCP and SCTP may be in order here.  In fact, I 
would suggest that at least one of Brian's statements about TCP behavior is 
in direct conflict with RFC793, as I explain below.   However, on a least 
one point, I would argue that Brian and Randall have come close to reaching 
consensus, although amid the smoke and flame it may be hard to perceive.

"Brian F. G. Bidulock" requested that SCTP be modified to be able to:
> > > > 2)  Cause the remote peer to stop transmitting new data.  RFC 2960
> > > >     permits an association to send new DATA regardless of whether
> > > >     it has receive a_rwnd = 0.

Randall Stewart proposed the following change:

>Note: Anytime the receivers rwnd is 0, the receiver MAY drop
>the DATA chunk without acknowledgment (since it has no room
>to receive the data nor validate it).

"Brian F. G. Bidulock" further noted:

>M2PA (and others) would like to use
>artificial lowering of a_rwnd to 0 as a way to stop the sender
>from sending new data or at least stopping transfer of new data.
>Yes, there is lots of real buffer space.  a_rwnd is for flow
>control and setting it to zero never means that all physical
>memory has been exahausted.  (You will see an ABORT very quickly
>after that happens).
>
>It appears that everything is devised to not require that
>data flow stop (as in TCP).  Who cares if some broken implementation
>that leaves a_rwnd = 0 and acks everything gets stuck?

However, RFC793 notes the following (section 3.7, p. 41)

>The sending TCP must be prepared to accept from the user and send at least
>one octet of new data even if the send window is zero.

Thus, in both TCP and SCTP it is possible to send new data on a zero 
advertised window.   So I think Brian's statement is in conflict with 
RFC793: the data flow does not "stop", if "stop" means that no new data can 
go on the wire.   However, in a larger sense, Brian raises a good point, 
because RFC793 continues:

>   ... The
>   sending TCP must regularly retransmit to the receiving TCP even when
>   the window is zero.  Two minutes is recommended for the retransmission
>   interval when the window is zero.  This retransmission is essential to
>   guarantee that when either TCP has a zero window the re-opening of the
>   window will be reliably reported to the other.
>
>   When the receiving TCP has a zero window and a segment arrives it must
>   still send an acknowledgment showing its next expected sequence number
>   and current window (zero).

So far, TCP and SCTP are in agreement at least to this extent... when presented
with a zero window, it is necessary for the sender to "probe" the window with
segments that may consist of new data.   A difference is that in TCP, these
window probes are a single byte, while in SCTP, they may consists of an entire
data chunk.

The key difference however, is that in TCP, as I understand it, the 
receiver will NOT accept
or acknowledge the NEW data (i.e. the window probe) if the advertised window
is still zero when the new data reaches the receiver.  While I could not 
find text in the
RFC that spells this out (can someone help here?) I did find a nice 
explanation and
example in W. Richard Stevens, TCP/IP Illustrated Vol 1, section 22.2, p. 
323-324.

If SCTP doesn't ALSO stubbornly refuse to acknowledge the new data that is sent
when the a_rwnd is zero, well, then I would agree that SCTP is broken in 
that regard.
If new data is sent when the a_rwnd is zero, then this new data should be 
considered
a "window probe".  The correct response to a window probe, if the a_rwnd is 
still zero,
should be to ignore the new data, and send a repeat of the previous 
acknowledgement
state ... the cum ack value, in the case of TCP, or a repeat of the 
previous SACK chunk
in the case of SCTP.

Now, it seems to me that this is exactly what Randall is proposing, though 
I would suggest
that the MAY should become a MUST, as shown in the square brackets:

>Note: Anytime the receivers rwnd is 0, the receiver MAY [MUST!] drop
>the DATA chunk without acknowledgment (since it has no room
>to receive the data nor validate it).

If the receiver DOES accept the DATA chunk
and ack it, where the heck is it supposed to go?    Unless the receiver has 
a "secret stash"
of memory beyond the advertised window, there isn't any place.  And, to NOT 
stop receiving
(so that, if the sender implements a blocking API, the backpressure can 
reach the sender)
is misleading to the sending application, and should be considered harmful.

Can we reach consensus on at least this ONE small point?

Phill Conrad

Phill Conrad -- Asst. Professor, CIS Dept, Temple University

NOTE: For faster replies, please reply to [email-masked], NOT 
to [email-masked].

_______________________________________________
tsvwg mailing list
[email-masked]
http://www1.ietf.org/mailman/listinfo/tsvwg

---

From: Kacheong Poon <[email-masked]>
Date: Mon, 5 Nov 2001 20:58:39 -0800 (PST)

I was shocked when I checked the tsvwg list...  I am not replying 
specifically to any mails (actually I cannot since there are so
many...), I just want to point out something since people are 
comparing SCTP and TCP's behavior regarding to zero window probe.
When I read the SCTP RFC, I always think that it is doing exactly
what TCP does and treat the outstanding chunk as zero window probe
when the data sender has a zero send window.  That's why I just skimmed
thru it.  I guess I missed something...  Please correct me if I am
wrong (and I may make some typoes...)  I want to clarify this mechanism
in TCP, in case there is misunderstanding (well, this is how I
understand it).

I think we should not just look at the a_rwnd in a packet itself.
We should consider the send window in the sender side and receive
window in the receiver side separately.  The reason is that it
is easier to deal with this conceptually, and they can actually
be out of sync (which is the whole reason why it needs a probe).  

Look at the TCP case first.  When the send window is zero, the 
sender cannot send anything.  But when every outstanding data
is ack'ed and the send window is still zero, TCP starts the zero
window probe mechanism.  Note that TCP implementation is not required
to have a persist timer (what BSD has) or a special timer for that.  It
is just an implementation detail.  A TCP stack can just use the
retransmission timer to trigger zero window probe.  That is the reason
why zero window probe contains new data (1 byte or MSS bytes for
some stacks).  The only difference than a real retransmission is that 
as long as there is new segment coming from the other side, it can be 
pure ACK segment or data segment, the retransmission timer firing to 
trigger zero window probe will not cause a connection timeout.

Note that TCP does not really care whether the data in the probe
is ack'ed or not, as all it cares during zero window probe period
is the send window.  If the data is ack'ed, TCP increases SND.UNA.
But if the send window is still zero, the probe mechanism continues.
The difference is that the data in the probe will be the next byte
of new data.

In the receiver side, the reason for having the sender to calculate
a zero window is that the app is not reading.  For most stack with
socket, it means that TCP has buffered SO_RCVBUF bytes of data.  Since 
the zero window probe is outside the receive window, TCP is required
to send back an ACK.  One tricky issue which may confuse people
is that there is also receiver side SWS avoidance mechanism.  What
it ensures is that TCP does not advertise silly small window.  The
interaction with zero window is that after the app reads some data
and TCP's buffer has fewer than SO_RCVBUF bytes of data, TCP
may still continue to make sure that the sender calculates a zero
send window.  Otherwise, the sender may send tons of small segments.
To simplify code in this case (no special check), most stacks will 
accept the 1 byte data in the probe because internally, the receive 
window is actually not zero.  This is OK as TCP still makes sure that
the sender's calculated send window is zero.

As people can see, it is a very simple mechanism.  And as usual, I
don't see why SCTP should behave differently.  That's why I skimmed
thru that part in SCTP RFC in the first place.  And I probably have 
missed it in the SCTP RFC, does it have a concept of SWS avoidance?  It
seems to me that it also applies to SCTP.

So in the SCTP case, the zero window probe is a new data chunk, which 
is only sent when the calculated send window is zero and there is no
outstanding unack'ed chunk.  This is specifically allowed in the spec.
It can be triggered using the T3-rtx timer.  As long as it will not 
timeout the association when there is new packet coming from the other 
side, it is fine.  One small point about triggering, it is network
and receiver side friendly not to send the probe immediately.  This
is one reason (other than avoiding complexity) that it is sent by T3-rxt
timer.  During the RTO while the sender is witholding sending new data 
chunks, the receiver may have cleared up the buffer and can send back an 
update so that the calculated send window is non-zero.  No probe is
needed.  And because of receiver SWS avoidance, the receiver may
send back a cum-ack (no gap ack because there should be no other
outstanding data chunk) which acks the window probe data chunk.  This
should be acceptable.  If the receiver does not have enough buffer space 
to hold the window probe data chunk, drop it and send back a SACK with 
the old cum-ack.  This makes sure that the sender will not time out the
association.

But if people think that accepting new data chunk above should not be 
allowed, then I think we need to specify that.  It means that if a receiver
is making the sender to calculate a zero send window, the receiver should
not accept new data chunks, regardless of its internal buffer space.
I think this is a small issue.

If the above zero window probe is not allowed (IMHO SCTP should behave 
like that), please explain why.  Or if SCTP does not behave like the above,
please explain the reasons and the difference.  I may have missed 
something.

On another note, it seems to me that people are suggesting that
SCTP should be allowed not to advance cum-ack but use gap ack to
sack received data chunk when there is actually no gap.  I may
have mis-read Randy's email, but it seems to me that he is doing
this with his BSD impl, maybe because he is experimenting copying out
to user space and doing checksum at the same time.  But I think
this is a hack and should not be allowed.  We need to specify that.
Unless we want to explicitly specify (which I think we should not)
in the spec that cum-ack is synchronously tied to ULP read, the above 
is a no no.  If it is not clear in RFC 2960, we should clarify
it in the new draft.

							K. Poon.
							[email-masked]

---

From: Randall Stewart <[email-masked]>
Date: Tue, 06 Nov 2001 06:07:14 -0600

Kacheong:

Comments below..

> I think this is a small issue.

The above description is exactly how I implemented it in the
FreeBSD/KAME kernel implementation we are building. Only thing
I am missing is the SWS code (which I will be adding).

Now, as far as the required to drop a chunk versus not. Brians
last email to me makes it clear what he wants. He wishes to
force the chunks to be dropped so that this mechanism can
be used as a flow control mechanism at the application 
layer. 

a) App sets rwnd (through API call?) to 0.
b) App reads anything left in queue.
c) Now app knows what to have resent on another association.

Of course a item that will happen unless the original assoc
is torn down is that

d) When rwnd is restored the old data will stream in.

Seems to me like a bad idea....

> is a no no.  If it is not clear in RFC 2960, we should clarify
No, I did not implement it this way on purpose.. I just don't
check for the error condition:

SACK(CUM-ACK=50, GAP=51,54);

As was later pointed out the above is a protocol violation and
is NON compliant to the spec... So I suppose I will go back
and write some screening code..

I am unsure what I should do when I see the above:

a) Abort the association?

or

b) collapse the SACK so that it logically appears
   as SACK(CUM-ACK=54);

Any suggestions from the list.. probably the abort would
be best since the peer may try to renege something considered
CUM-ACK'd... but I am open to suggestions.

R

-- 
Randall R. Stewart
[email-masked] 815-342-5222 (cell phone)

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Tue, 6 Nov 2001 06:41:47 -0600

Randall,

> d) When rwnd is restored the old data will stream in.

And be flushed...

> CUM-ACK'd... but I am open to suggestions.

Consider the case where there are GAPs and the implementation
advances the Cum Ack but fails to detect that the first gap
has closed before generating a SACK.  In this case it might be
better to collapse the SACK (only because the sender failed to)
rather than aborting the association.

--brian

-- 
Brian F. G. Bidulock    ¦ The reasonable man adapts himself to the ¦
[email-masked]    ¦ world; the unreasonable one persists in  ¦
http://www.openss7.org/ ¦ trying  to adapt the  world  to himself. ¦
                        ¦ Therefore  all  progress  depends on the ¦
                        ¦ unreasonable man. -- George Bernard Shaw ¦

---

From: Jacob Heitz <[email-masked]>
Date: Tue, 06 Nov 2001 08:46:51 -0800

A third option is to allow it.

Delete the CUM-ACKed data from the retransmission buffer.
Don't delete the GAP-ACKed data (just yet), but don't
retransmit it either (for now).

---

From: Kacheong Poon <[email-masked]>
Date: Tue, 6 Nov 2001 15:12:25 -0800 (PST)

> a) App sets rwnd (through API call?) to 0.

This is another tricky issue.  But first, reducing the receive
window may be done via, assuming socket API, setsockopt(SO_RCVBUF).
In TCP, it is strongly encouraged (need to check RFC 1122 to see
if it is a MUST or SHOULD) that receiver does not shrink its window. 
Shrinking the window means that some data in flight will be discarded
by the receiver.  Besides wasting network resources, the sender
may be confused and depending on the new window size, data in the
retransmit queue (those data which is discarded by the receiver 
because of the shrinked window) may cause the sender to time out
the connection in the worst case.  I think this also applies to
SCTP.

So the only safe way for an ULP to cause SCTP to tell the sender 
side's SCTP to stop sending is to stop reading data from SCTP.  
Then SCTP will buffer the unread data and eventually the sender
will stop sending.  But I'm afraid this may not be what sigtran
folks want.  Please correct me if I misunderstand their need.

> Seems to me like a bad idea....

I guess sigtran folks, say Brian, need to state clearly what
they really want and we can see if SCTP can help them.

> CUM-ACK'd... but I am open to suggestions.

I think this behavior should be clarified in the draft so that 
no implementation, hopefully, will do this.  But to me,
it seems that aborting the association if SCTP sender detects
this is quite drastic.  I think transports normally are quite 
accommodating (-:  But collapsing them also means removing those
chunks from the retransmit queue.  This is probably against
the spec as gap ack does not allow that.  Can we just use the 
normal code to handle this?  This automatically means that if 
the receiver does not advance the cum-ack (because of a bug or 
something), the sender will eventually stop sending.  Note that 
I am not encouraging people to violate the spec.  

So IMHO, (b) should not be done.  (a) is better but seems too
drastic.  Maybe we don't do anything special is a good choice 
too (-:

							K. Poon.
							[email-masked]

---

From: Randall Stewart <[email-masked]>
Date: Tue, 06 Nov 2001 18:18:52 -0600

> drastic.  Maybe we don't do anything special is a good choice

I think you are right here.. besides it takes one more item
off my board to fix in the implementation :->

Being forgiving in what you accept is probably a good thing... 
besides, as you say, since cum-ack is not moving the association
will get gated by all these gap-ack's and data will slow
down to a crawl since the cwnd will not be allowed to
open up... this in itself should be a deterent.. i.e. you
heard your own throughput by doing this...

It is an automatic penalty that shows up (the best type)!

R

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Tue, 6 Nov 2001 18:22:08 -0600

Kacheong,

> the connection in the worst case.  I think this also applies to

Yes, this was the intended use: to artificially choke (I wouldn't
say shrink because it could go from a very large value to 0
instantly).  I agree that it is a bad idea for an application to
do this.  Application-level flow control is much better in that
it does not dork with rather critical SCTP parameters.

> folks want.  Please correct me if I misunderstand their need.

That won't work for M2PA.  It needs to STOP traffic flow abruptly.
Application-level flow control would work better.

> they really want and we can see if SCTP can help them.

Is it not clear?  M2PA needs to STOP traffic flow abruptly and
withhold acknowledgements for a period of seconds and then
either abort the association (leaving data for retrieval at the
sender) or resume traffic flow at previously experienced levels.

Artificially setting a_rwnd = 0 was seen as a potential (but not
necessarily advisable) way of doing this.

Thus the query to TSV.

Because of some serious concerns on the impact of these two
approaches on SCTP operation, I think M2PA (and SCTP) would
be far better off if M2PA implemented ULP-level flow control
with its own sequence numbers and acks.

--brian

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Tue, 6 Nov 2001 18:56:24 -0600

Randall,

> heard your own throughput by doing this...

cwnd might be wide open already.  Doing this does not
hurt cwnd.  Even if congestion is ocuring in the middle
of this, cwnd just doesn't grow until something is Cum
acked.  With a_rwnd = 0, it is worse: idle -> slow start.

--brian

---

From: Chip Sharp <[email-masked]>
Date: Wed, 07 Nov 2001 00:04:25 -0500

Be liberal in what you receive and strict in what you transmit.

What harm will it do to treat the Cum-Ack and Gap Ack as normally defined 
in the spec for the receive side?

Chip

At 07:07 AM 11/6/2001, Randall Stewart wrote:
...snip...