Subject: Re: [Tsvwg] SCTP: Data chunk in flight when a_rwnd = 0

From: Vern Paxson <[email-masked]>
Date: Sat, 03 Nov 2001 23:42:48 -0800

> Have I missed somewhere in the RFC where it mentions that
> the receiver can refuse to acknowledge received DATA chunks
> containing new data when it is advertising a receive window
> of 0?
> 
> Or was it the intention of SCTP that it not permit a
> receiver to stop traffic by setting a_rwnd = 0?

The intention was that the sender is always allowed to *try* sending a
bit of data beyond the window to serve as a probe to see if the window
has in fact opened up, but that also the receiver can indeed stop traffic
by setting the advertised window to 0.

The sender might not have previously-transmitted data lying around with
which to make its attempt, so it's allowed to do it with new data.  The
receiver does not have to accept this data; if it doesn't, it should send
a SACK describing what data it has received & accepted, along with a window
advertisement.

That said, the above is my recollection of the intent when we added the
rule in question - I haven't looked through the RFC to see how it has been
written up.

		Vern

_______________________________________________
tsvwg mailing list
[email-masked]
http://www1.ietf.org/mailman/listinfo/tsvwg

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Sun, 4 Nov 2001 02:13:46 -0600

Vern,

On Sat, 03 Nov 2001, Vern Paxson wrote:

> which to make its attempt, so it's allowed to do it with new data.

Actually, one doesn't need new or old data, just an already
acked TSN.  A duplicate TSN will have the effect you are
looking for without needing the old data (whatever data is
included must be discarded).

> The

Although this makes, sense, the RFC does not permit the receiver
to refuse acknowledgement of new DATA whether a_rwnd = 0 or not
(Section 6.2).  Nor is it described anywhere that the sender of
these probing packets is permitted to stop pegging association
and path retransmit counters, even if the other side does withhold
acknolwedgements.

> rule in question - I haven't looked through the RFC to see how it has been

Take a look.  I don't think that the current text in the RFC
achieves this intent in several respects:

    1.) It does not permit the endpoint to withhold acks of
        new data when a_rwnd = 0 as you describe.  (Section 6.2
        says that every new DATA chunk must be acknolwedged and
        sets upper bounds on the time limit for acknowledgement.)

    2.) It does not permit the endpoint to send SACK immediately
        when receiving new data while a_rwnd = 0 as you describe.
        (Section 6.2 says that the enpoint should only SACK
        immediately on duplicates, gaps, or to update a_rwnd, and
        should ack every second DATA chunk otherwise.)

    3.) When "probing" in this fashion, the transmitter is still
        required to run RTO and perform cwnd calculations also
        to peg association and path retransmits.  This could lead
        to failure of the association (depending on heartbeat
        intervals) if the receiver is withholding acks in the
        fashion that your describe.

Although I agree that it should be the intention, the current
text in the RFC certainly does not have the effect of stopping
traffic flow.  The above three things need to be corrected first:
the receiver advertising a_rwnd = 0 must be permitted to withhold
acknolwedgements, required to SACK immediately, and the transmitter
probing a peer adversiting a_rwnd = 0 should not perform congestion
or retransmission calcaltations on the "probes".

Rather than changing all those things in the RFC, it would probably
be better to take the opposite interpretation and say that the
transmittter MUST NOT send new data (as the text says now).  And
that any probe chunk must be with an already acknowledged TSN.
This will have the effect that any data along with the TSN will be
discarded (and could be bogus), the receiver will be forced to
SACK the duplicate TSN immediately, and new data will not flow.

Comments?

-- 
Brian F. G. Bidulock    ¦ The reasonable man adapts himself to the ¦
[email-masked]    ¦ world; the unreasonable one persists in  ¦
http://www.openss7.org/ ¦ trying  to adapt the  world  to himself. ¦
                        ¦ Therefore  all  progress  depends on the ¦
                        ¦ unreasonable man. -- George Bernard Shaw ¦

---

From: Randall Stewart <[email-masked]>
Date: Mon, 05 Nov 2001 06:12:47 -0600

Vern:

Your recollection is dead on.. as to the issue that
Brian raises I will get out the RFC and do a bit
of study this AM... and send a subsequent message.

R

-- 
Randall R. Stewart
[email-masked] 815-342-5222 (cell phone)

---

From: Randall Stewart <[email-masked]>
Date: Mon, 05 Nov 2001 06:30:24 -0600

Brian:

Has promised I have looked at the RFC and in some ways you
are correct... in one sense I could say you are not...

It does state (as you kindly pointed out):

"
   The SCTP endpoint MUST always acknowledge the reception of each valid
   DATA chunk.
"

But I ask you one question..

If your a_rwnd is 0 and you have no space for the data have you
really received the data? 

I can see some arguing.. well yes of course I received it the
interface got a m_buf and gave it to the SCTP stack.. but there
is a counter argument that says ... hey there is no room
so we don't even do the CRC check but instead just drop the
packet... 

I can see adding a clearification in the implementors guide
(which I need to re-issue anyway)... Something like:

"
Note: Anytime the receivers rwnd is 0, the receiver MAY drop
the DATA chunk without acknowledgment (since it has no room
to receive the data nor validate it).
"

R

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Mon, 5 Nov 2001 08:25:53 -0600

Randall,

This is not helping us.  M2PA (and others) would like to use
artificial lowering of a_rwnd to 0 as a way to stop the sender
from sending new data or at least stopping transfer of new data.
Yes, there is lots of real buffer space.  a_rwnd is for flow
control and setting it to zero never means that all physical
memory has been exahausted.  (You will see an ABORT very quickly
after that happens).

It appears that everything is devised to not require that
data flow stop (as in TCP).  Who cares if some broken implementation
that leaves a_rwnd = 0 and acks everything gets stuck?

Not being able to rely on this characteristic may require the
use of an application-level sequence number, acknowledgements,
and flow control window so that the application can in no
uncertain terms stop data flow.  In M2PA, stopping data flow as
a precursor to changeover in the event of processor busy and
processor outage conditions is required to curtail message loss.

An implementation that does not stop acking while advertising
a_rwnd = 0 will break this.

Will SCTP support M2PA in this regard or does N2PA need to provide
new a new transport protocol?

(SSCOP-MCE does not have these problems.)

--brian

---

From: Randall Stewart <[email-masked]>
Date: Mon, 05 Nov 2001 09:17:35 -0600

Brian:

I am not sure I understand the object here... maybe another
tweak that follows below would make you more comfortable...

"
Note: Anytime the receivers rwnd is 0, the receiver MAY drop
the DATA chunk without acknowledgment of the new DATA
chunk (since it has no room to receive the data nor validate it).
However the receiver MUST still send an acknowledgment showing
the new chunk as still missing with a r_wnd update folloing 
the delayed acknowledgment guidlines specified below.
"

I fail to see how if the DATA is new or not is that big of
issue. If you are allowed to drop it silently but you
still must send a SACK. The association will stay up, the
peer will get the r-wnd update (hopefully there is one) and
you will eventually retransmit the dropped data.

R

---

From: Chip Sharp <[email-masked]>
Date: Mon, 05 Nov 2001 12:28:01 -0500

A few thoughts:

The spec says the sender can have *one* DATA chunk in flight, regardless of 
rwnd.

If the sender reaches the point that rwnd=0 due to outstanding DATA chunks 
that have not been SACKED, then it will stop transmitting since DATA chunks 
are already in flight.  If the sender receives a SACK with a_rwnd=0 and the 
SACK does not acknowledge all DATA chunks in flight, then the sender can't 
send any more DATA chunks based on the above rule.

Thus, the only thing the sender has control over is whether or not it can 
send 1 DATA chunk when rwnd=0 and there are no outstanding DATA chunks. If 
the sender doesn't probe for a change in rwnd, it will never be able to 
transmit new data (deadlock).  If the receiver does not acknowledge this 
DATA chunk, the sender cannot transmit any more DATA chunks since the DATA 
chunk is "in flight" until acked.

If the receiver does transmit a SACK with a_rwnd=0, the receiver will still 
receive new DATA chunks that were in flight prior to the sender receiving 
the SACK.  Thus, the receiver must be able to deal with the situation in 
which it receives new DATA chunks after sending a_rwnd=0 regardless of the 
operation of the sender.

The only problem seems to be whether or not the receiver acknowledges DATA 
chunks received after it sends out a_rwnd=0.
      * No:  The receiver will receive a periodic retransmission of one of 
the missing DATA chunks on timeout.
      * Yes: The receiver will receive a new DATA chunk per round-trip time 
as each single new DATA chunk is acked.

Chip

>SACK the duplicate TSN immediately, and new data will not flow.
...snip...

---

From: Qiaobing Xie <[email-masked]>
Date: Mon, 05 Nov 2001 13:52:54 -0600

Brian,

There may be room for further clarification in RFC2960 on how the
receiver should handle new data with its a_rwnd=0, but I don't see
anything wrong. 

After (artificially or not) setting a_rwnd to 0, all the receiver needs
to do when new data arrives is to drop the new data and send back an
SACK repeating the current cumulative TSN. This way, the data flow is
effectively stopped because the sender is forced to keep retransmitting
the same new data over and over. This should achieve what you are
looking for in M2PA.

-Qiaobing

---

From: Randall Stewart <[email-masked]>
Date: Mon, 05 Nov 2001 14:30:51 -0600

Q:

And some wording clearifications will be made in the
next impl-guide so that should settle that one.. it
will not fix the per-stream flow control which I think
is really what he is after...

R

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Mon, 5 Nov 2001 14:55:53 -0600

Qiaobing,

M2PA does not have control over the SCTP implementation.  If the
SCTP implementation acks data after advertising a_rwnd = 0, it
will be unsuitable for M2PA.  How does M2PA state that the
suitability of SCTP depends upon its implementation and not
the RFC?  If the RFC says MUST NOT cumulatively ack new data
when a_rwnd = 0, M2PA can be assured that any correct SCTP
implementation will be suitable and can recommend SCTP as
a transport.  Otherwise, we are back where we started.

--brian

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Mon, 5 Nov 2001 15:07:00 -0600

Chip,

> send any more DATA chunks based on the above rule.

But the receiver is not allowed to not acknowledge DATA chunks...

> transmit new data (deadlock).

It can send new data when the receive opens up a_rwnd.  There is only a 
deadlock situation if a SACK is lost (the one that opened up a_rwnd).
Thus the need to force a SACK or two out of the receiver ever so often.
DUP segments in TCP do this just fine.  DUP TSNs in SCTP would work as
well (they force an immediate SACK which either contains a_rwnd = 0 or
some other value).

> If the receiver does not acknowledge this DATA chunk, the sender cannot
> transmit any more DATA chunks since the DATA chunk is "in flight" until
> acked.

But if the receiver is required (or permitted) to ack the DATA chunk,
DATA will flow.

> which it receives new DATA chunks after sending a_rwnd=0 regardless of the 

This is not the required behavior.  It is required that acks of data in
flight also be withheld once a_rwnd has been advertized zero.  Otherwise
each of these DATA chunks will be lost to the network with no copy in
the sender's retransmission buffer for retrieval and retransmission on an
alternate association.

> chunks received after it sends out a_rwnd=0.

No data chunks can be acknowledged after it asserts a_rwnd = 0, it doesn't
matter when it hits the line or gets to the other side.

> as each single new DATA chunk is acked.

Plus the data chunks in flight which are not to be acked.  Each of these
messages is a message lost to SS7 with M2PA.  MTP3 conformace requires that
no messages (0, zero, nicht, nil) messages be lost in these changeover
scenarios.  One message gone astrasy and MTP3 cannot conform over M2PA.

If SCTP stops cum acking on a_rwnd = 0 assertion all is well.  A MAY does
not do M2PA any good here...

--brian

---

From: Qiaobing Xie <[email-masked]>
Date: Mon, 05 Nov 2001 15:21:20 -0600

Chip,

Please see my comments in-line..

-Qiaobing

> send any more DATA chunks based on the above rule.

Yes. 

And there is a chance that the still-in-flight chunks will eventually
fill up some gaps at the receiver, and as a result free up some resource
causing the a_rwnd to become open again in the next SACK.

> chunk is "in flight" until acked.

It does not make sense for the sender NOT to probe if it has retrans or
new data pending. Not to send data while it can is a protocol violation
in itself. The receiver also must answer with an SACK but not to ack the
current DATA chunk in 200 ms (per the delayed SACK rule). This way, the
cumulative TSN will not move and the sender will retransmit the chunk
again (i.e., probing again).

> which it receives new DATA chunks after sending a_rwnd=0 regardless of the

Yes, receiver will: 1) drop the new DATA 2) follow the delayed SACK rule
and send back an SACK after 200 ms.

> as each single new DATA chunk is acked.

I think the delayed SACK rule applies here since there is no gap. If the
receiver does have a gap and sets the a_rwnd=0 at the same time, then
the responsible implementation should be able to avoid this by invoking
the renege and partial delivery rules to get rid of the gap before
setting a_rwnd to 0.

---

From: Qiaobing Xie <[email-masked]>
Date: Mon, 05 Nov 2001 15:26:46 -0600

"Brian F. G. Bidulock" wrote:
> will be unsuitable for M2PA.  

Brian,

All this says is that M2PA needs to use a good SCTP implementation,
which is what M2PA as well as any other upper layers should do.

-Qiaobing

> How does M2PA state that the

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Mon, 5 Nov 2001 15:43:43 -0600

Qiaobing,

> which is what M2PA as well as any other upper layers should do.

Err, with curent wording, it needs an incorrect SCTP implementation.
6.2 does not allow a correct implementation to withold acks.

--brian

---

From: Randall Stewart <[email-masked]>
Date: Mon, 05 Nov 2001 17:25:42 -0600

> But the receiver is not allowed to not acknowledge DATA chunks...

I fail to understand why you continue to beat a dead horse? I have
said that the next implementors guide.. which IS a supplement to
the RFC and the MUST's etc in this WILL apply just as forcefully
as RFC2960 (when we are done with it) .. will have the wording
to allow the receiver to drop the chunk.. 

Continuing to rant about something that is going to be fixed
and will be available in the implementors guide draft is
non-productive and a waste of everyones cycles..

> well (they force an immediate SACK which either contains a_rwnd = 0 or
And allowing a receiver to DROP a segment due to lack of
buffer space and SACK with no update does the same thing. This
is similar to what TCP does. Yes some implementations do send
dup data.. others do not... the spec will be made similar..

> But if the receiver is required (or permitted) to ack the DATA chunk,

see above..

> the sender's retransmission buffer for retrieval and retransmission on an

You make no sense above. If data is dropped (due to lack of buffer
space) and SACK are sent duplicating, if you will, the last SACK
sent aka same cum-ack, a-rwnd-0 + whatever gap-ack blocks still
present... It will keep the sender happy... all is well AND
it will let the receiver clear its buffer problem.

> not do M2PA any good here...

Either a implementation will STOP accepting new data and cum-ack
the way things were when rwnd=0 occured, and thus NO DATA is
lost... OR the implementation will pull the DATA in and SACK it..

In either case NO data is lost... 0, zero , nicht, nil...

R

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Mon, 5 Nov 2001 18:05:11 -0600

Randall,

> to allow the receiver to drop the chunk.. 

There is a difference between "allow" and "require".  Allow may
fix SCTP's glaring problem.  It would take Require to meet M2PA's
needs.

> it will let the receiver clear its buffer problem.

Again it is the difference between "allow" and "require".

If we can say that SCTP will not support application-level flow
controls and that the ULP needs to institute their own, I would
be happy to drop the matter and leave it at "allow".

> lost... OR the implementation will pull the DATA in and SACK it..

In which case the data is lost to MTP3 resulting in conformance
failure.  If TSV will say "don't use SCTP this way" then we can
procedure to institute M2PA sequence numbers, acks and retransmission.

> In either case NO data is lost... 0, zero , nicht, nil...

You are not looking at redundancy across associations from the
ULP perpsective.  The data is lost to the system because it is
on the wrong side of the association.

--brian

---

From: Randall Stewart <[email-masked]>
Date: Mon, 05 Nov 2001 20:24:03 -0600

> procedure to institute M2PA sequence numbers, acks and retransmission.

One of two things will happen here... all ending the same way

A) If the implementation drops the data, the SACK's would 
   continue to be sent without reporting the data dropped.
   Eventually the rwnd condtion will clear.. and guess
   what the data is retransmitted.

or

B) The implementation lied and had a bit more buffer space
   and does not drop the data but sacks it... still reporting
   a zero rwnd. In stevens it is mentioned how BSD's do this
   now with TCP. 

In either case the data IS DELIVERED to the upper layer eventually..

resulting in my statement below..

> on the wrong side of the association.

Huh, what do you mean wrong side of the association. No matter
what happens it all arrives at the receiver. We have only 
one association here... If you are doing things with two
parallel associations that is outside the scope of this
discussion. 

The end result is going to be the same, when the rwnd
goes back to positive the data will either get retransmitted
OR the data will already be there when the reader starts
reading again..

R

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Mon, 5 Nov 2001 21:47:58 -0600

Randall,

> parallel associations that is outside the scope of this

That is precisely the purpose for M2PA, to provide reliability
*between* associations with MTP3 performing the changeover.
I'm pretty sure I made this clear from the start.  At least 
several of us have been discussing just that in this thread.

What did you think M2PA needed it for?

--brian

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

From: Randall Stewart <[email-masked]>
Date: Tue, 06 Nov 2001 05:51:56 -0600

> What did you think M2PA needed it for?

Ok so what you are attempting to do is have two
seperate SCTP associations between say different
sets of IP addresses.

Then use the 0 rwnd case to stop the flow of
data (I guess for flow control). 

Now a couple of observations:

1) This is not what SCTP flow control in an association
   was designed for. I.e. the concept is to stop sending
   data and then restart because you want it all to arrive
   on the single association.

2) When the application says rwnd=0 there will always be
   data in flight (at least in theory) and some that
   is inqueue now that as yet, not been read.

3) You must deal with the as not been read data anyway if
   you were to attempt to twist the rwnd=0 for this mechanism

4) The data in flight will arrive within 1/2 RTT.

5) You could just use method (3) to deal with the data still
   arriving for 1/2 RTT.. which, depending on the app, you may
   still be processing what is in queue anyway for this time.

6) If this data is all ending up on the same machine why the
   association does not span all of its IP addresses is a puzzel
   to me..

R

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Tue, 6 Nov 2001 06:12:07 -0600

Randall,

>    on the single association.

Ok.

>    is inqueue now that as yet, not been read.

inqueue is under control of implementation.  Implementation
can ensure that no data is queued.

>    you were to attempt to twist the rwnd=0 for this mechanism

MTP deals with this by not queueing beneath MTP3.  This is
all under the implementation's control.

> 4) The data in flight will arrive within 1/2 RTT.

And must not be acknowledged.

>    still be processing what is in queue anyway for this time.

Processing has stopped.

>    association does not span all of its IP addresses is a puzzel

They are not the same machine.  These associations go
to separate hosts.

--brian

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Tue, 6 Nov 2001 06:50:14 -0600

Randall,

On Tue, 06 Nov 2001, Randall Stewart wrote:

> Comment below...
> Ok, now I have a clear picture of what you are doing...
> 
> a) You still have an issue, even if we changed the MAY
>    you object to, to a MUST. The issue is that you WILL
>    get the DATA when you set rwnd back to some positive value.
>    This means all that DATA you are trying to stop is going
>    to all of a sudden show up.

The sending implementation is completely in control of this.
It can flush, call back data, whatever it wants.  The procedure
calls for moving this data to the new association.  How it is
called back from SCTP is an implementation matter.

> 
> b) If you are then sending to the other machine (via some
>    mechanism ) that terminates the parallel association then
>    you must tell it.. hey the last SSN I got was X. How will
>    you know when you do a_rwnd=positive, if the data first
>    out is old or new data. 

Take a look at Q.703 and Q.704 Clause 5 if your interested.

> 
> c) You state above that the queue is under control of the
>    implementation. Ok, I can buy that, but then in that
>    case you also have control of the implementation to 
>    make sure that the MAY is a MUST. 

The local implementation cannot control the implementation
at the other end of the association.  That is what the RFC
is for.  If I go MUST and the other end goes MAY, the 
combination will fail conformance.

--brian

---

From: Randall Stewart <[email-masked]>
Date: Tue, 06 Nov 2001 06:35:07 -0600

Comment below...

R

---

From: David Lehmann <[email-masked]>
Date: Tue, 06 Nov 2001 08:22:24 -0500

Randall,

The first "acknowledgment" seems to refer to the cum-ack or
gap report in the SACK.  The second "acknowledgment" seems to refer
to the SACK itself.  Is that correct?  If so, please consider
changing the second (and third?) "acknowledgment" to "SACK".
(Also, the next paragraph in the RFC may need the same consideration.)

BTW, I think "acknowledgment" and "acknowledgement" are both
valid spellings, but "acknowledgement" is used in the RFC.

-David