Subject: [Tsvwg] u-sctp comments

From: "Armando L. Caro Jr." <[email-masked]>
Date: Mon, 23 Jul 2001 06:30:31 -0400 (EDT)

Hi,

I have a few comments/suggestions about the unreliable data mode extension
draft...

I think the draft should provide explicit detail about handling the T3-rtx
timer when attempting to retransmit due to a timeout or a fast
retransmit. Also, any data chunk which is marked as acked by exhausting
the allowed number of retransmissions must not credit partial_bytes_acked.

I suggest the following changes:

Section 4.2.A2.B3
"
     B3) If the "Unrel.Trans.Count" is currently 0, the sender MUST NOT
         retransmit the data chunk. Instead, the sender MUST mark the
	 data chunk as being finally acked.
"
changed to:
"
     B3) If the "Unrel.Trans.Count" is currently 0, the sender MUST NOT
         retransmit the data chunk. Instead, the sender MUST mark the data
         chunk as being forward acked and thus ineligible for crediting
         partial_bytes_acked. In addition, the sender MUST no longer
         consider the data chunk outstanding. The sender MUST stop the
         T3-rtx timer if there are no more oustanding data
         chunks. Otherwise, the sender MUST restart the T3-rtx timer.
"

The added text about the T3-rtx timer _could_ be inferred from the fact
that the data chunk is no longer considered outstanding, but I think
without explicitly mentioning it, implementations may overlook this subtle
point. While testing the unreliable option in our ns-2 implementation, I
have noticed that if this point is overlooked, a burst loss may suffer
from a spurious timeout.

Section 4.2.A3
"
 A3) whenever the data sender receives a SACK from the data receiver,
     it SHALL first process the SACK using the normal procedures as
     defined in Section 6.2.1 of [RFC2960].
"
changed to:
"
 A3) whenever the data sender receives a SACK from the data receiver,
     it SHALL first process the SACK using the normal procedures as
     defined in Section 6.2.1 of [RFC2960], with the following
     exception. Any data chunk marked as forward acked MUST not credit
     partial_bytes_acked.
"

Other suggestions....

Section 4.2 - "Unrel.Trans.Count" should be "Unrel.Retrans.Count"

Section 4.2
"
   Note, if a TSN is indicated as missing by a SACK carrying gap reports
   AND the TSN is earlier than the current "Advanced.Peer.Ack.Point", the
   data sender MUST NOT take any action on this TSN, ...
"
should be:
"
   Note, if a TSN is indicated as missing by a SACK carrying gap reports
   AND the TSN is earlier than or equal to the current
   "Advanced.Peer.Ack.Point", the data sender MUST NOT take any action on
   this TSN, ...
"

I know as far as RFC2119 is concerned, SHALL = MUST. However, I find
interchanging these words throughout the document somewhat
confusing. Wouldn't it be better to simply pick one (say, MUST), and use
it religiously throughout the document?

Thanks,
Armando

-----------------------------------------------------------------------
Armando L. Caro Jr.                                  [email-masked]
University of Delaware                   http://www.cis.udel.edu/~acaro
-----------------------------------------------------------------------

_______________________________________________
tsvwg mailing list
[email-masked]
http://www1.ietf.org/mailman/listinfo/tsvwg

---

From: Randall Stewart <[email-masked]>
Date: Mon, 23 Jul 2001 11:46:48 -0500

Armando:

In general great comments... just a few quick comments
and I am sure Qiaobing may have more :)

>          chunks. Otherwise, the sender MUST restart the T3-rtx timer.

This looks good but I am not sure about the 

"mark the data chunk as being forward acked and thus..."

I understand what you mean, but is it clear for other readers .... :-)

>      exception. Any data chunk marked as forward acked MUST not credit

This makes sense being hand and hand with the previous change..

> Section 4.2 - "Unrel.Trans.Count" should be "Unrel.Retrans.Count"

Yes, I think that is better..

> it religiously throughout the document?

Hmm... I think we should change all SHALL to MUST ... I did not
realize we had any "SHALL's" in the document ... sigh...

R

-- 
Randall R. Stewart
[email-masked] 815-342-5222 (cell phone)

---

From: Qiaobing Xie <[email-masked]>
Date: Mon, 23 Jul 2001 11:59:55 -0500

Armando,

I think all your comments below are excellent and I agree to all of them
and will incorporate them into the next revision.

thanks,
-Qiaobing

---

From: "Armando L. Caro Jr." <[email-masked]>
Date: Tue, 24 Jul 2001 02:45:20 -0400 (EDT)

> I understand what you mean, but is it clear for other readers .... :-)

Yes, I know... but that is the best wording I could think of. Perhaps
someone can suggest a better wording.

---

From: "Armando L. Caro Jr." <[email-masked]>
Date: Thu, 26 Jul 2001 10:23:09 -0400 (EDT)

Hi,

I have some modifications/additions to my previous suggestions. Plus, I
have some new comments...

Section 4.2.A2.B3
"
     B3) If the "Unrel.Trans.Count" is currently 0, the sender MUST NOT
         retransmit the data chunk. Instead, the sender MUST do the
         following:

	    i) Consider the data chunk no longer oustanding and decrement
	    the number of outstanding bytes to reflect this.

            ii) Mark the data chunk as being forward acked and thus
            ineligible for crediting partial_bytes_acked or decrementing
            the number of outstanding bytes.

	    iii) Stop the T3-rtx timer if there are no more oustanding
	    data chunks. Otherwise, restart the T3-rtx timer.
"

Section 4.2.A3
"
 A3) whenever the data sender receives a SACK from the data receiver,
     it SHALL first process the SACK using the normal procedures as
     defined in Section 6.2.1 of [RFC2960], with the following
     exception. Any data chunk marked as forward acked MUST not credit
     partial_bytes_acked or decrement the number of outstanding bytes.
"                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

The additional changes above are intended to ensure that an implementation
modifies the number outstanding bytes at the time the FOWARD TSN is
sent. Without being explicit, an implementation may choose to wait for the
new cum ack (which reflects the FORWARD TSN) before decrementing the number
of outstanding bytes. I have noticed that waiting for the new cum ack must
not be done because in some situations, a deadlock can occur... the sender
is waiting for a SACK to opens up cwnd, while the receiver is waiting
for a DATA chunk to generate a SACK. This all happens simply because the
sender has an incorrect notion of the number of outstanding bytes. Once
the sender gives up on rtx'ing a tsn, it should immediately update this value.

Section 4.2:
"
   Note, if a TSN is indicated as missing by a SACK carrying gap
   reports... When this happens, it is normally an indication that a
   previous FORWARD TSN from the data sender may have been lost in
   the network.
"
should be:
"
   Note, if a TSN is indicated as missing by a SACK carrying gap
   reports... When this happens, it is normally an indication that a
   previous FORWARD TSN from the data sender may have been lost in the
   network or may have arrived at the data receiver after the transmission
   of another SACK with the TSN missing report.
"

The reason for the added text is that the data sender may transmit a
FORWARD TSN in response to a SACK, but there may already be SACKs in
flight which also report the same TSN missing. Hence, the sender should
recognize the fact that although these SACKs are arriving at the data
sender after the FORWARD TSN was transmitted, the SACKs in flight can not
possibly reflect the FORWARD TSN. This leads me to my next point...

There is a similar problem with FORWARD TSN chunks as there was with Fast
Rtx (which was addressed in the implementor's guide). The problem is that
once the first FORWARD TSN chunk is sent, there may already be other SACKs
in flight. Each of these will cause the sender to transmit yet another
FORWARD TSN chunk.Jana and I have spent some time thinking of a
solution, but it appears that there isn't one as straight forward as there
was with Fast Rtx. You can't simply say that FORWARD TSNs are to be
transmitted only once, because what happens if it gets lost? Then you
would need acks for FORWARD TSNs, which is NOT what we want! One possible
remedy (not solution) is to apply the delayed sack concept to FORWARD
TSNs, but perhaps others have a better alternative??

We have encountered a problem which is not specific to u-sctp, but seems
to only arise (in our simulations at least) when u-sctp is used. The
problem is with respect to timeout rtx's. When a timeout occurs, the
sender retransmits as many outstanding DATA chunks that can fit in one MTU
size packet. Suppose the following scenario:

     - one stream in the association and it is unreliable
     - TSN x has a k-rtx value of 1 (ie, will be rtx'd at most once)

Now suppose the following sequence of events:

     1. TSN x is dropped
     2. TSN x gets fast rtx'd, but that gets dropped too
     3. timeout on TSN x, but sender doesnt rtx it
     4. sender packs the latest TNS(s) into the rtx packet
     5. receiver receives rtx packet, but contains duplicate TSN(s)

So why does the receiver get duplicate TSN(s)??  Well since TSN x is the
only dropped TSN, the only outstanding data chunks are those which have
been sent most recently and have not been given enough time to be
acked. Again, we have no solution to suggest. Any ideas?

---

From: Qiaobing Xie <[email-masked]>
Date: Thu, 26 Jul 2001 20:38:23 -0500

Armando,

See my comments in-line....

-Qiaobing

>      partial_bytes_acked or decrement the number of outstanding bytes.

I agree with you in principle that the above details can be useful to
the implementor. But, as you have also pointed out, all these are
implied by the data sender "marking the chunk as finally acked". Your
text above is good but a little too "codifying" the design, I probably
will "soften" the wording a little bit when I pull it into the draft.

> possibly reflect the FORWARD TSN. This leads me to my next point...

The above para is only for explanatory purpose (i.e. not specifying any
action), but it seems to cause more confusion than helping the reader. I
am actually thinking about removing it since not to react to a missing
TSN report behind your cumTSN point is kind of normal thing to do no
matter you are u-sctp or sctp.

> TSNs, but perhaps others have a better alternative??

I am not even sure whether I will call this a problem :-) This is just a
common kind of race condition (you get a FwdTSN and SACK criss-crossing
each other in flight). Before talking about solution or remedy, what is
the consequence or cost because of this? I don't think there is much.

> acked. Again, we have no solution to suggest. Any ideas?

So, if TSN x is just a reliable normal chunk, this situation will still
be very similar - the receiver will get one good retransmitted chunk
(TSN x) and all the other will be useless dups. 

For this to happen, you need to have TSN x being first sent in a
different packet from the rest of the chunks and being bundled with the
rest of chunks on the second transmission, since otherwise, they all
would have been lost together in the first packet :-) I'd say, when
re-bundling and loss are happening at the same time, it is reasonable
for the receiver to see some dups.

This scenario is also implementation specific, i.e., if your
implementation is running a rtx timer per chunk, then this won't happen.

---

From: "Armando L. Caro Jr." <[email-masked]>
Date: Fri, 27 Jul 2001 06:05:07 -0400 (EDT)

Qiaobing,

My responses are inline...

> will "soften" the wording a little bit when I pull it into the draft.

Well, I did think that it may be too "codifying", but I thought it may be
important to give the implementor less freedom here.

> matter you are u-sctp or sctp.

This is true... ok, agreed.

> the consequence or cost because of this? I don't think there is much.

Well at some point, delayed ACK/SACKs were determine to be a good thing
because there were too many on the line. This was determined with TCP,
whose pure ACKs were 20 bytes. Forward tsns are only 8 bytes, but I don't
think this makes it less of an issue.

In the case of constant traffic, the number of forward tsns sent for a
single loss is related to the current cwnd. So even if an isolated loss
happens only occasionally, there will be many fwdtsns on the line.

> (TSN x) and all the other will be useless dups.

Not unless re-bundling happens... I agree that if loss and re-bundling
happen at the same time, then duplicates are reasonable. With u-sctp,
however, timeouts on chunks w/ k-rtx value of 1 will almost always cause a
dup.

> implementation is running a rtx timer per chunk, then this won't happen.

ok, but how common is this? Are implementations actually maintaining
timers per chunk?

---

From: Qiaobing Xie <[email-masked]>
Date: Fri, 27 Jul 2001 12:17:10 -0500

Armando,

...
> think this makes it less of an issue.

Size difference is one thing, but the really fundamental difference
between SACKs and FwdTSNs is that the number of SACKs is proportional to
the data you are sending (in the worst case every TSN can trigger a
SACK), while FwdTSNs are only sent when you have packet losses. This
means that, during normal operation, the more data you send the more
SACKs you will see and the more you should concern about the bandwidth
consumed by SACKs. This is why we have the delayed SACK rules. On the
other hand, the number of FwdTSNs you will need to send has little to do
with the amount of data you send. For a normally operating association,
the packet loss ratio should be fairly small and therefore you should
only see a few number of FwdTSNs.

If the number of FwdTSNs over an association becomes large enough to
make you start concerning about the bandwidth they consume, you've
probably got some much bigger problems with the association :-)

> happens only occasionally, there will be many fwdtsns on the line.

You need to explain this further. I don't seem to understand your
analysis here. To me, you only need to send one FwdTSN if an isolated
loss happens.

> ... With u-sctp,

How is so? And why is k-rtx = 1 so unique? 

I still believe this can only happen _if_ you rebundle the chunks, and
u-sctp makes little difference. 

> ok, but how common is this? Are implementations actually maintaining

I don't know how common it is, but for those who'd like to avoid sending
duplicates at time-out this becomes a sensible alternative to consider
:-)

-Qiaobing

---

From: Randall Stewart <[email-masked]>
Date: Fri, 27 Jul 2001 12:55:00 -0500

Some comments in line... 

"Armando L. Caro Jr." wrote:
snip.....snip......snip............

> ok, but how common is this? Are implementations actually maintaining

If I remember right from the email thread:

TSN(x - U)------->Xlost
TSN(x+1 - R)----->
TSN(x+2 - R)----->
...
TSN(x+3 -R)------>
TSN(x+4 -R)------>
FR TSN(x -U)---->Xlost

Time-Out(TSN x)

Now in this situation the problem is that we must assume that
TSN x+5 and up may be inflight and transmitted some time after
our failed FR of TSN(x). Now I am wondering in this situation
if it would not be better to NOT retransmit anything but to
restart the T3-Timer on TSN(x+5)... Or alternatively if
the implementation was keeping transmit time deltas (in lieu
of a timer per chunk.. I think a timer per chunk will be
a rare thing ... like you) one could restart the timer for
TSN(x+5) with the delta difference between the TSN(x)'s last
retransmit time and the time TSN(x+5) (or whatever one is
next to be sent, if not TSN(x+5)) timestamp...

Otherwise as you say, sending useless retransmits may occur. Of
course these would only occur if you had a drastic change in
RTT time or lost SACK's since the peer should be SACKing every
packet in this case... i.e. its got a gap between TSN(x) and
TSN(x+5). This may be a corner case not worth worrying about.
Maybe your simulations can give us an idea as to how often
this would occur?

R