Subject: Re: [Tsvwg] More on PRSCTP-00

From: Qiaobing Xie <[email-masked]>
Date: Mon, 03 Jun 2002 14:18:52 -0500

Brian,

I see what you are getting at now. This is another issue that was
intensely discussed off-line before. It is really a tradeoff between
design simplicity and more efficiency. See more comments below.

"Brian F. G. Bidulock" wrote:
> 
> Qiaobing,
> 
> Well, that's not really what I was gettin at: consider just the
> receiver's view of this sequence:
> 
> <-- TSH   SID  SSN
> 
>     101    1    2
>     102    1    3
>     103    2   57
>     104    2   58
> 
>     FORWARD TSN 106
> 
>     108    1    5
>     109    1    6
>     110    2   60
>     111    2   61
> 
> Now, because the receiver cannot tell whether SID/SSN (1/4) was
> dropped or whether SID/SSN (2/60) was dropped (until it receives the
                              ^^^^^
       you must meant to say (2/59) here

> missing TSN 107), missing TSN 107 has blocked both streams.  If more
> streams were involved in the same way, they would all be blocked by
> the loss of one TSN.  This *is* inter-stream HOL blocking.
      ^^^^^^^^^^^^^^^
This is a misstatement. As you also came to the realization below, you
must lose >2 TSNs across both streams for the above scenario to happen
(in your example, you assumed the loss of 3 TSNs across both streams).
If you only lose one TSN, it is impossible to affect more than 2 streams
and thus there is *no chance* for any inter-stream HOL blocking.

> The problem is not as great as I first thought (all streams
> blocked), really it is just streams that have dropping that are
> blocked, but the problem is still of concern: particularly as it is
> congestion which causes drops in PR-SCTP in the first place, one can
> expect packet losses at about the same time.

It is not totally unreasonable to let a stream suffer some msg delivery
delay when the stream is missing a TSN in the first place.

> 
> The delay introducted is blocking (in the example), delivery on
> stream 1 and stream 2 until TSN 107 can be finally and successfully
> delivered.

It is so because you actually lost TSN 105, 106, and 107 to start the
scenario and each of the 2 stream lost at least one TSN, otherwise it
would be affected.

> 
> If FORWARD TSN included the fact that, say, stream 1 should "expect"
> SSN 5 and stream 2 should "expect" SSN 60 (because 4 and 59 have
> been dropped), then there would be no blocking.

Yes, it will help if we include SSN info in the Fwd-TSN, but the
question is how to encode it. Your suggestion here does not scale well
because, say, if we have 10k outbound streams, how can you encode the
"expected" SSN for each one of them in a Fwd-TSN?

An alternative might be to enlist in the Fwd-TSN each "dropped" chunk
with its TSN/SID/SSN triplet. But this means that we will still "burn" 8
bytes for each dropped data chunk, while one of the primary ideas of not
to re-send expired data is to save network bandwidth...

Maybe this info can be provided as optional, i.e., the Fwd-TSN will
always indicate the new cumTSN position, and also OPTIONALLY provide a
list of TSN/SID/SSN triplets to help the receiver message delivery. This
can even be a partial list if too many TSNs are dropped/skipped since it
will still help the receiver somewhat...

-Qiaobing

> 
> In general I don't see a problem with putting the SID/SSN in the
> FORWARD TSN.  Certainly the sizeof of the FOWARD TSN chunk could be
> limited in similar ways to SACK chunks (advisory only), then the
> approach appears to be a sound one.
> 
> --brian
>

_______________________________________________
tsvwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tsvwg

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Mon, 3 Jun 2002 14:43:42 -0500

Qiaobing,

>        you must meant to say (2/59) here

Yes, of course.

> > the loss of one TSN.  This *is* inter-stream HOL blocking.

That is not true, especially for SIGTRAN where DATA chunks are quite small.
Loss of one SCTP packet could easily result in the loss of 32 TSNs at once
making interstream blocking between 33 streams quite possible and even
probably in this situation.

> delay when the stream is missing a TSN in the first place.

That's not the issue, the issue is suffering a delay when a TSN has been
dropped by PR-SCTP and not as a result of packet loss.  If dropping the
packet does not boost performance from the application's perspective, why
drop packets?

> scenario and each of the 2 stream lost at least one TSN, otherwise it

The example for 33 streams is a valid one as well.  It is actually quite
probably that packet loss will occur around the same time as dropping,
because congestion leads to both loss and dropping under "timed reliability".

> "expected" SSN for each one of them in a Fwd-TSN?

It is only necessary to code the "Expected SSN" for streams which have
dropped TSNs.  Consider it as a Forward-SSN.  It should never be necessary
to code 10k streams but could be treated as truncation on GAP reports
nevertheless.

> to re-send expired data is to save network bandwidth...

Sending Forward-SSN would only take about 8 bytes and would be included
no more than once per stream dropping for no more than 8 bytes per dropped
packet.  The DATA chunk header is 16 bytes, so you will be saving quite a
few bytes.

> will still help the receiver somewhat...

Sending a Forward-SSN for only those streams that drop ordered packets
within the cumm-ack to Forward-TSN range would require far less data
in the FORWARD TSN, yet provide only what is required to deliver around
drops, takes 4 bytes less data than your triplet approach, and could still
be truncated when required.  I would say MUST, because this is as important
to SCTP's ability to avoid this interstream blocking as are GAP reports.

--brian

-- 
Brian F. G. Bidulock    ¦ The reasonable man adapts himself to the ¦
[email-masked]    ¦ world; the unreasonable one persists in  ¦
http://www.openss7.org/ ¦ trying  to adapt the  world  to himself. ¦
                        ¦ Therefore  all  progress  depends on the ¦
                        ¦ unreasonable man. -- George Bernard Shaw ¦

---

From: "Armando L. Caro Jr." <[email-masked]>
Date: Mon, 3 Jun 2002 16:19:01 -0400 (EDT)

On Mon, 3 Jun 2002, Brian F. G. Bidulock wrote:

> be truncated when required.

So you are proposing a new forward tsn chunk like the following?

         0                   1                   2                   3
         0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
        +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
        |1 1 0 0 0 0 0 0|  Chunk Flags  |0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0|
        +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
        |                      New Cumulative TSN                       |
        +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
        |      Num of Forward SSNs = N  |         ??                    |
        +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
        |      Stream Identifier S      |   Stream Sequence Number n    |
        +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
        /                                                               /
        \                              ...                              \
        /                                                               /
        +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
        |      Stream Identifier S      |   Stream Sequence Number n    |
        +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

> I would say MUST, because this is as important to SCTP's ability to
> avoid this interstream blocking as are GAP reports.

I think MUST is good, otherwise PR-SCTP may not provided much better
performance than plain SCTP.

~armando

0--                                                --0
| Armando L. Caro Jr.     |       [email-masked] |
| University of Delaware  |  www.cis.udel.edu/~acaro |
0--                                                --0

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Mon, 3 Jun 2002 15:38:29 -0500

Armando,

Yes!  Maybe the "??" field could be used to indicate the total
number of dropped TSNs (of course maxing out at 0xffff) reported
by this FORWARD TSN chunk.  The receiver might be able to use that
to determine whether it has accounted for all drops with the
provided Forward SSNs.

The procedure would be upon receipt of a FORWARD TSN chunk to perform
the cummulative ack adjustments already provided in the draft, however,
the next expected stream sequence number for each inbound stream would
be adjusted forward to the SSN contained in the Forward SSNs.

--brian

---

From: "Armando L. Caro Jr." <[email-masked]>
Date: Mon, 3 Jun 2002 16:44:23 -0400 (EDT)

> to determine whether it has accounted for all drops with the

yeah, _maybe_

> be adjusted forward to the SSN contained in the Forward SSNs.

Yeah, this is the same idea we had... but yours is better in that we don't
have to report foward ssns for all streams... only those that need it.

~armando

0--                                                --0

---

From: Qiaobing Xie <[email-masked]>
Date: Mon, 03 Jun 2002 20:13:49 -0500

.....
> That is not true, 
   ^^^^^^^^^^^^^^

Brian,

I am sorry but IT IS TRUE! I am merely pointing out the fact that if you
lose just one TSN then there won't be an inter-stream HOL blocking as
you wrongly stated. Do you agree with this point?

> especially for SIGTRAN where DATA chunks are quite small.

This is an entirely different point than what I was talking about. Yes,
if the app tends to bundle a number of small messages from different
streams in a single package, then multiple streams will be
affected/delayed if you lose the packet. But this is always the case,
regardless you use PR-SCTP or not. We all know this for a long time and
PR-SCTP is not designed to address this issue.

In reality, let's say you bundle 32 TSNs from different strms in a
packet and it gets lost in the network, and at the time of retrans,
let's assume that, of the 32 TSNs, 10 have expired. With SCTP your
sender will re-send all the 32 TSNs, probably bundled into another
packet. With PR-SCTP, you will send a Fwd-TSN to skip the 10 expired
TSNs if you could, and re-send the other 22 TSNs that are still valid,
and you would very likely bundle your Fwd-TSN and the 22 TSNs into a
packet. In such a case, PR-SCTP would never cause any extra delay than
SCTP.

....

> performance than plain SCTP.

This is a big misunderstanding of the original intention of PR-SCTP (and
that of U-SCTP and the primitive "lifetime" feature in SCTP). We didn't
introduce this feature as a way to "boost performance from the
application's perspective". Instead, this was motivated by the question
"why we keep retransmitting expired user messages and waste the network
bandwidth?"

Just think about it, when a lost TSN(s) is detected, the only difference
between a PR-SCTP sender and a SCTP sender is that the former sends a
Fwd-TSN and the latter re-send the lost TSN(s). In either case the lost
TSN problem will take a one-way network delay to resolve (assuming that
the Fwd-TSN or the re-send will not be lost again). There is no time
advantage between the both. But there is a clear advantage for PR-SCTP
in terms of saving in the network bandwidth (however, if we keep
inflating the size of Fwd-TSN, this advantage will become diminished).

I'd agree that there may be some circumstantial advantages for PR-SCTP,
but even if they exist, they are mostly secondary and case-dependent.

In summary, I still don't see clearly how PR-SCTP introduces extra delay
to the delivery of inter-stream messages. Comparing to what SCTP does,
PR-SCTP does not do any better, nor does it do any worse in this
department. But this is ok to me, since it is mainly designed to improve
network bandwidth efficacy by not retransmitting expired user data. 

-Qiaobing

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Mon, 3 Jun 2002 21:59:52 -0500

Qiaobing,

> > > and thus there is *no chance* for any inter-stream HOL blocking.

No I do not agree: consider the receiver's view of the following:

  <--- TSN 101  SID 1  SSN 1
  <--- TSN 102  SID 2  SSN 1
  <--- TSN 103  SID 3  SSN 1
  <--- TSN 104  SID 4  SSN 1
  <--- TSN 105  SID 5  SSN 1
  <--- TSN 106  SID 6  SSN 1
  <--- TSN 107  SID 7  SSN 1
  <--- TSN 108  SID 8  SSN 1
  <--- TSN 109  SID 9  SSN 1

  <--- FOWARD TSN = 118 (9 TSNs dropped)

  <--- TSN 120  SID 1  SSN 3
  <--- TSN 121  SID 2  SSN 3
  <--- TSN 122  SID 3  SSN 3
  <--- TSN 123  SID 4  SSN 3
  <--- TSN 124  SID 5  SSN 3
  <--- TSN 125  SID 6  SSN 3
  <--- TSN 126  SID 7  SSN 3
  <--- TSN 127  SID 8  SSN 3
  <--- TSN 128  SID 9  SSN 3

Here *loss* of a single TSN has blocked 9 streams.  The receiver
cannot know if TSN 119 contains SSN 2 for any of the 9 streams.
None of the SSN 3's for any of the nine streams can be delivered
until TSN 119 has been successfully retransmitted.  This is a severe
blocking situation.

If you consider the use of a "Forward SSN" then you have something
like:

  <--- TSN 101  SID 1  SSN 1
  <--- TSN 102  SID 2  SSN 1
  <--- TSN 103  SID 3  SSN 1
  <--- TSN 104  SID 4  SSN 1
  <--- TSN 105  SID 5  SSN 1
  <--- TSN 106  SID 6  SSN 1
  <--- TSN 107  SID 7  SSN 1
  <--- TSN 108  SID 8  SSN 1
  <--- TSN 109  SID 9  SSN 1

  <--- FOWARD TSN = 118 (9 TSNs dropped)
       SID 1 Forward SSN = 3
       SID 2 Forward SSN = 3
       SID 3 Forward SSN = 3
       SID 4 Forward SSN = 3
       SID 5 Forward SSN = 3
       SID 6 Forward SSN = 3
       SID 7 Forward SSN = 3
       SID 8 Forward SSN = 3

  <--- TSN 120  SID 1  SSN 3
  <--- TSN 121  SID 2  SSN 3
  <--- TSN 122  SID 3  SSN 3
  <--- TSN 123  SID 4  SSN 3
  <--- TSN 124  SID 5  SSN 3
  <--- TSN 125  SID 6  SSN 3
  <--- TSN 126  SID 7  SSN 3
  <--- TSN 127  SID 8  SSN 3
  <--- TSN 128  SID 9  SSN 3

Then it is obvious to the receiver that SSN 3 can be delivered
around the lost packet for SIDs 1 thru 8, but that SSN 3 for SID 9
must await retransmission of TSN 119.  (Because in the example it
must contain SID 9 SSN 2).

4 bytes per streams which has dropped packets in the FOWARD TSN is
then no more consuptive of packet payload than GAP reports in SACKs.

> PR-SCTP is not designed to address this issue.

There is nothing in SCTP RFC 2960 operation which keeps a non
PR-SCTP implementation from delivering SSNs around lost TSNs
regardless of user data size or bundling.  In th above case,
_except_for_the_effect_of_dropped_packets_in_PR_SCTP_ SCTP would
normally delivery the next sequential SSN in the stream around
missing TSNs.  So what you way above does not ring true to me.

> packet. In such a case, PR-SCTP would never cause any extra delay than

Yes it would, due to the effect above.  The dropped PR-SCTP TSNs
break the normal increment by one rule of the SSN that regular SCTP
uses to determine when TSNs from one stream can be delivered around
TSNs lost for other streams.  It is only because regular SCTP
implicitly knows that the next expected SSN in an ordered stream is
one greater than the last received SSN which allows SCTP to do this.
Regardless of lost TSNs, I know that I have the last received SSN+1
in  an ordered stream that it can be delivered regardless of missing
(lost or reordered) TSNs.  When a receiver receives a FORWARD TSN,
it breaks this rule that SSNs must increase by one, temporarily
destroying regular SCTP's ability to avoid HOL blocking by
delivering around lost TSNs.  I'm sorry but PR-SCTP degrades normal
SCTP performance in this regard.  Placing a Forward SSN in the
FORWARD TSN chunk for those streams which have had SSNs dropped,
restores this relationship.

> "why we keep retransmitting expired user messages and waste the network

But please not at the expense of reintroducing HOL blocking and
making the application perform worse under PR-SCTP than under
regular SCTP.  There is no need to do this at the application's
unnecessary expense.

> inflating the size of Fwd-TSN, this advantage will become diminished).

The saving in the network might be in terms of TSNs, but the
degradation to the application is because of the poor treatment of
SSNs: the abrogation of the rule that SSNs must increase by one.

> but even if they exist, they are mostly secondary and case-dependent.

This is true for GAP reports as well.  I think that experience with
GAP reports has shown that the implementation can quite efficiently
make use of GAP reports to save network resources while still
permitting the impelmentation to avoid HOL blocking.  The same
should easily be true for Forward SSNs, they permit the application
to continue to avoid HOL blocking will allowing the network
efficiency of dropping stale packets rather than retransmitting
them.

> network bandwidth efficacy by not retransmitting expired user data. 

Take a really close look at the example above.  Consider where
65,536 streams have just each dropped the next ordered packet because
of a timer expiry, and one TSN is subsequently lost.  All 65,536
streams are blocked from delivering the next TSN to the ULP until the
single (yes one) lost TSN is retransmitted.

Again, consider that this behavior is not a corner case: the same
congestion conditions which cause "timed reliability" dropping also
cause packet loss in non-ECN networks.  The congestion which caused
a the packet drops ("timed reliabilty" expiry) is apt to drop some
next packet due to the same congestion.

--brian

---

From: Qiaobing Xie <[email-masked]>
Date: Tue, 04 Jun 2002 19:14:16 -0500

....

Brian,

Huh, I see now why we are having problem understand each other's
points... we are using different terminology:

When I said "dropped TSNs", I spoke from the receiver's perspective and
referred to TSNs the receiver sees as missing in the TSN space. While
you use "dropped TSNs" to refer to those that are assigned to data
chunks but never sent out (and you use "lost TSNs" to refer to those
that are put on the wire and then lost in the network).

With this clarified, I'd say your above example is more than a corner
case - it is actually very strange. It basically shows that your data
sender assigns 65535 TSNs (one for each stream) to messages in the
sender's outqueue and then changes its mind and never put those 65535
TSNs on the wire.

I'd say that to a certain degree this is equivalent to that the sender
intentionally create the 65535 holes in the TSN space (but somehow still
expects the receiver to perform). This example in fact illustrates the
importance of the note in RFC2960 that the sender should always try to
postpone the assignment of TSNs to the last moment. A good
implementation should expire messages that have never been transmitted
without burning holes in the TSN space. Fixing holes in the receiver's
TSN space (with either plain retransmission of base SCTP or Fwd-TSN of
PR-SCTP) will always cost you. One would think the sender should always
try its best not to create holes in the TSN space. 

-Qiaobing

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Tue, 4 Jun 2002 19:50:20 -0500

Qiaobing,

> that are put on the wire and then lost in the network).

Yes.  PR-SCTP talks of TSNs being "dropped".  I use the terminology
from the PR-SCTP draft when referring to "dropped" TSNs.

> sender's outqueue and then changes its mind and never put those 65535

The example is from the receiver's view.  It says nothing about whether
the sender placed the TSNs on the wire or not.  It may have, and they
may have been "lost" yet expired (timed reliability) before retransmission
resulting in the FORWARD TSN.  This is not strange, it is even highly
likely if the "timed reliability" interval is about 4 RTT.

> try its best not to create holes in the TSN space. 

Sorry, but these TSNs may have been sent by the sender (meaning that
TSNs were assigned and cannot be deferred any longer), but expired before
successful retransmission.  In the example of 9 streams under times of
congestion, it is quite possible that 9 TSNs were lost in a single SCTP
packet, or even in multiple retransmission, the FORWARD TSN got thru
(because it was small), and subsequent TSNs transmitted on the stream are
also "lost".

There is no assumption of a poorly designed sender in the example.

This is a problem: a real problem, easily addressed by the Forward SSN
approach.

--brian

---

From: Qiaobing Xie <[email-masked]>
Date: Wed, 05 Jun 2002 15:26:43 -0500

Brian,

> congestion, it is quite possible that 9 TSNs were lost in a single SCTP
                                        ^^^^^^^^^^^^^^^^
> packet, or even in multiple retransmission the FORWARD TSN got thru

Then, your (repeated) statement (citing one of them here):

> > Here *loss* of a single TSN has blocked 9 streams.
    ^^^^^^^^^^^^^^^^^^^^^^^^^

is untrue (or at least grossly mis-stated). 

> This is a problem: a real problem, easily addressed by the Forward SSN

I've said at the very beginning that I am not "violently" against the
idea of adding SSN info to Fwd-TSN. All I've been trying to point out to
you is that your scenario in which multiple streams can be blocked can
only result from the loss of multiple TSNs. This is not a problem
induced by PR-SCTP, the same thing can happen with the base SCTP. 

To make myself clear:

- Is the problem as serious as you claimed to be? 
No. You overstated it. 

- Whether we will be able to alleviate this problem by adding some
complexity to PR-SCTP design?
Yes, as I've already said that adding SSN to Fwd-TSN can have benefits. 

- Whether the added complexity worth the benefit? 
Maybe or maybe not, I am a doubter but I will not violentely object it
if others want to do it.

-Qiaobing

---

From: Qiaobing Xie <[email-masked]>
Date: Wed, 05 Jun 2002 15:30:48 -0500

Hi, Randy,

Pls see my latest reply to Brian...

-Qiaobing

Randall Stewart wrote:
> 
> Brian/Qiaobing:
> 
> I have read through this interesting debate. Qiaobing and
> I had a major discussion on this way back when it was
> considered U-SCTP. Renee Revis first pointed this out to me.. but
> from a different persepective i.e. the receiver being able to
> efficently process stuck TSN's ...
> 
> After a lengthy debate I agreed to not have the SSN/SID listed
> in the FWD TSN... I did NOT think of the HOL blocking issue.. or
> I would have used it as a issue (I was the one arguing for the
> SSN/SID in the FWD-TSN :->).
> 
> After watching this debate.. I think I have come to the conclusion that
> Brian is correct :)
> 
> Michael R. is here... I will discuss it with him.. I want to get
> the other co-author's take on this one... If the others agree with
> Brian (and me :>) then I think we will put this in.... also I would
> like to hear what the rest of the WG feels on this???
> 
> I think it is reasonable and not that much overhead to include JUST
> the dropped streams NEXT expected SSN... You would be adding 4 bytes
> for EVERY stream dropping one or more TSN's... note that this does
> NOT mean 4 bytes per TSN but 4 bytes per where one or more TSN's were
> dropped..
> 
> I think this is reasonable.. it improves processing AND avoids HOL
> blocking..
> 
> Michael T/Phil? what do you think... Michael R.. I will talk to you at
> the
> next class break..
> 
> R
> 
> --
> Randall R. Stewart
> [email-masked] 815-342-5222 (cell phone)

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Wed, 5 Jun 2002 16:35:02 -0500

Qiaobing,

> > congestion, it is quite possible that 9 TSNs were lost in a single SCTP

Pardon, let us say loss of a single SCTP packet.
It is still HOL blocking.

--brian

---

From: Qiaobing Xie <[email-masked]>
Date: Wed, 05 Jun 2002 17:00:36 -0500

Phil,

Please take my little challenges below as a way to help you making your
case tighter :-)

Phillip Conrad wrote:
...
> The next thing that happens is that TSNs 110 through 118 are "dropped" in
> the technical sense of that word; see the defn in the current draft.  This
> means that the sender gives up on sending them, and issues a forward
> TSN.   However, no Forward TSN is issued for TSN119.  This may be because
> the messages on stream 9 are treated as Reliable.   This is NOT an unusual
> situation; 

What unusual here is that you seem to imply that you send the Fwd-TSN
without retransmitting the *reliable but lost* TSN119. Remember, TSN119
is lost *together* with the other 8 TSNs, and the decision point of
sending Fwd-TSN is *deliberately* crafted to be the same as that for
normal retransmission. 

Therefore, what should be usual is that when the sender reached the
conclusion that it needs to send the Fwd-TSN for the 8 "dropped" TSNs,
it MUST also have reached the conclusion at the same moment to
retransmit TSN119 (furthermore, I'd think in more implementations the
Fwd-TSN will also be piggybacked with the retrans TSN119).

The stream blocking analysis (by Brian) only holds if the Fwd-TSN
arrives without the retransmitted TSN119. But if both arrive at the
same, would the analysis still hold?

> ...We have all the PR streams
> potentially being HOL blocked by loss of a single packet.  I think we need
> to fix this.

Interesting point. But we seem to accept the same thing (streams
potentially being HOL blocked by loss of a single packet) in base SCTP.
How come?

regards,
-Qiaobing

---

From: Qiaobing Xie <[email-masked]>
Date: Wed, 05 Jun 2002 17:10:07 -0500

> It is still HOL blocking.

Again, what's really funny is that we suddenly start to see this HOL
blocking caused by the loss of a single SCTP packet (presumably
containing multiple TSNs from different streams) as an issue. We seem to
be able to live with it in the base SCTP for a long time.

-Qiaobing

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Wed, 5 Jun 2002 17:18:26 -0500

Qiaobing,

> same, would the analysis still hold?

The draft allows the FORWARD TSN to be sent immediately upon dropping and
there is no requirement to bundle retransmitted TSNs with the FORWARD TSN.

> potentially being HOL blocked by loss of a single packet) in base SCTP.

We don't.

--brian

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Wed, 5 Jun 2002 17:15:56 -0500

Qiaobing,

> be able to live with it in the base SCTP for a long time.

But these TSNs were also dropped.  It is the further loss of a single
TSN which causes the blocking.  It is not the case that the loss of a
single TSN from one stream blocks other streams, reagardless of the history
leading up to the loss of the TSN.

I'm afraid you're on your own on this one.

--brian

---

From: Qiaobing Xie <[email-masked]>
Date: Wed, 05 Jun 2002 17:35:36 -0500

...
> I'm afraid you're on your own on this one.

Sadly but that seems to be the case :-) 

After all, democratic consensus building is meant to bring what the
*group as a whole* views as the best. But, make no mistake, we are *not*
the same as what's called "designed by committed". We are better than
"them" :-)

-Qiaobing

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Wed, 5 Jun 2002 18:07:15 -0500

Randall,

On Wed, 05 Jun 2002, Randall Stewart wrote:

> A few comments..
> 
> Phillip Conrad wrote:
> > 
> > Dear Colleagues:
> >       I agree with Randy that Brian has raised an important issue for us to
> > consider, and I think has explained it quite well with a very clear,
> > illustrative example.
> > 
> > The question is one of a tradeoff between two designs:
> > 
> > (A) current design: simpler, fewer bits on the wire, but may (re)-introduce
> > an HOL blocking problem
> 
> I must disagree with your "it is simpler" ... it is NOT. The question is
> where is the burden. Having implemented pr-sctp in the kernel of freebsd
> NOT having the stream number/SSN in the FWD-TSN means I must keep
> an extra set of linked lists and such to track any stream that has
> pending stuck data. When a fwd-tsn arrives I must HUNT all of these
> streams and look at each inbound queue.
> 
> The counter to this is the sender (in Brian and my earlier proposal
> that was to Q only) must track this info to send with the FWD-TSN.
> Actually
> less of a burden IMHO...

I agree.  I discovered this situation while implementing PR-SCTP in the
Linux Kernel.  The normal receiver can rely on the fact that the next
expected SSN is the last fully received SSN plus 1.  PR-SCTP without
a forward SSN causes the receive to have to update this expected value
with the next SSN in the reorder queue whenever there are no TSN gaps.
This even might require that when TSN gaps are filled that the receiver
walk all inbound stream structures to update the next expected SSN.
In an independent implementation, I have arrived at the same conclusion:
that the reciever has more work to do for FORWARD TSN without a forward
SSN.

In fact the receiver has this work to do if it has ever received a
FORWARD TSN.  This is because it is not possible for the receiver to
know a priori which streams could be affected by this scenairo.  Therefore,
the load on the receiver is increased even when this case does not occur.
It is the mere possibility of its occurence which introduces the load.
(Each time gaps close, the next expected SSN for all streams need to be
re-examined whether there are SSN gaps or not.)

On the sender side, the sender must always track the last sent SSN and
assign the next SSN (++SSN).  Indeed the sender when generating a FORWARD
TSN chunk would need to traverse outbound stream structures with dropped
TSNs to formulate Forward TSNs, but this is likely to be a far shorter
list than at the receiver and non-PR streams would never need to be
involved.

--brian

> > 
> > (B) alternative design: more complex, may require larger Forward TSN
> > chunks, but avoids re-introducing an HOL blocking problem, and thus
> > improves performance in some situations.
> 
> 
> see above it is NOT more complex... 
> 
> > 
> > The question of whether to choose A or B depends on whether you believe
> > that the situation Brian describes is a "corner case" or not.   So I've
> > walked through Brian's example, and have determined that I don't think it
> > is a corner case at all.   So, I'm inclined to agree with Randy and Brian
> > that we should consider a change.
> > 
> > But I'd like to invite others to walk through my line of reasoning to test
> > whether I've reasoned this correctly.
> 
> 
> I agree it is NOT a corner case..
> 
> > 
> > My rationale that Brian's example is NOT a corner case:
> > ========================================
> > 
> > In Brian's example, messages are being sent round-robin on a number of
> > streams.  This seems like a fairly reasonable use of SCTP streams, do that
> > aspect does not strike me as unusual.
> 
> I agree :>
> 
> > 
> > Now for the next part of this explanation, I'm going to have to infer
> > certain things, so, Brian, please correct me if I infer wrongly.   *One*
> > possibility for the sequence of events leading to the
> > situation Brian describes is that there is a burst loss of data chunks
> > TSN110 through TSN119 (ten chunks, but possibly less than ten packets..
> > perhaps even only ONE packet if the chunks are small enough to be
> > bundled!)... and then data chunks TSN120 through TSN128 make it through
> > OK.   This does not strike me as particularly unusual; a loss of a single
> > packet followed by a successful packet is sufficient to cause this situation.
> > 
> 
> Or the other possiblility would be a set of bundled chunks in one packet
> being lost.. either way is the same result I would think..
> 
> > situation; it is perfectly reasonably to assume that an application might
> > have "k" streams with data that can be dropped, and "n - k" streams of data
> > that must be treated as reliable.  In the case of Brian's example, n is 10,
> > and k is 9, but I think that his example will still produce similar results
> > even for other values of "k" and "n".
> > 
> 
> Note.. streams themselves are not reliable or partially reliable. With
> the new draft it is the message itself which is sent with the
> reliablilty
> set to "fully reliable". One possilbe implementation would be to make
> things correlate to a particular stream.. but this is just an
> implementation
> option..
> 
> > Bottom line:
> > ========
> > 
> > I don't think its a corner case at all.   We have all the PR streams
> I agree..
> > The solution of forward SSNs is one viable solution I think.  There may be
> > others, but that's as good as any for right now I think.   There may be
> > another solution as well, but let me do some writing and thinking about it
> > off line before I muddy the waters with it here... just to make sure it is
> > a viable alternative.
> 
> agreed
> 
> 
> > 
> > Randy... I'm willing to make the text changes and send a proposed version
> > around to the co-authors for comment if there is consensus to move in that
> > direction.   Let me know.
> > 
> 
> I will get it out this weekend if I may.. there are some interesting
> things that I have thought about that must be covered... I had something
> like this in an old u-sctp draft (during Q and I's debate earlier :>)
> 
> 
> 
> > 
> > (Side note: I was worried that the "technical definition" of the word
> > "dropped" would cause confusion, and it already has.   Perhaps should use
> > another word in the draft to describe the operation where the sender
> > decides to no longer transmit and/or retransmit a specific data chunk.  How
> > about the term "abandoned" in place of "dropped"?)
> 
> 
> Sounds good..
> 
> note, Michael R also agrees .. or seems to .. we will talk more..
> 
> R
> 
> 
> > 
> > /ptc
> > 
> > At 11:44 AM 6/5/2002 -0500, Randall Stewart wrote:
> > >Brian/Qiaobing:
> > >

---

From: "Armando L. Caro Jr." <[email-masked]>
Date: Wed, 5 Jun 2002 19:28:43 -0400 (EDT)

> improves performance in some situations.

(A) is actually more complex from the receiver's point of view. (B) gives
the receiver explicit SID/SSN information about "dropped" TSNs.

> I don't think its a corner case at all.   We have all the PR streams

No, it isn't.

> about the term "abandoned" in place of "dropped"?)

I agree that "dropped" is somewhat ambiguious... abandoned is a better
term.

~armando

0--                                                --0

---

From: Qiaobing Xie <[email-masked]>
Date: Wed, 05 Jun 2002 18:57:31 -0500

R,

...
> No, I don't think you can make this assumption.. see above... only
> if TSN119 was bundled with the rest...

Well, I didn't make this assumption, but Phil (and Brian) did. If you go
back to read Phil's email, *he* built his case assuming the 9 TSNs are
bundled in a single packet. This is why I said the case Phil used to
illustrate the problem can use some tighten-up :-)

...
> But a single loss can never block more than ONE stream.. I think
> this is the difference...

The argument from Brian and Phil is based on a single loss of a packet
that contains multiple TSNs from different streams. I am arguing if this
were to happen to the base SCTP, all streams that own one or more TSNs
in the lost packet will be affected in the same way.

Anyway, I think you guys have already made up your mind on this... I
have no objection as said before.

-Q

---

From: Qiaobing Xie <[email-masked]>
Date: Thu, 06 Jun 2002 16:27:20 -0500

Hi, Phill,

Phillip Conrad wrote:
...
> Let me see if understand your point.. I think you are saying that from a
> holistic standpoint, the overall system may be simpler if we introduce
> forward SSNs, because as you, Armando and Brian have all pointed out, the
> additional fields can actually make implementation of the sender and
> receiver more straightforward, and more efficient... i.e., simpler.   I can
> accept this line of reasoning.

To add more bits to the wire in exchange for some implementation ease is
a valid point. I will always buy this argument, as long as we don't go
to the extreme.

....
> The objection that Brian raises, and others (included myself) have
> concurred with, is that the FORWARD TSN chunk, as specified in the current
> draft, creates the possibility that this principle may be negated. As
> Brian's example illustrates, the loss of a TSN unrelated to stream j may
> delay delivery of packets on stream j, because when a FORWARD TSN moves the
> cum ack point forward, there is an ambiguity about which streams the
> "skipped" packets belong to. The FORWARD SSN corrects this ambiguity.

I have tried very hard but failed to draft up an "all considered"
scenario with 2 streams to show how one missing TSN from one stream can
unfairly block the delivery of a queued, ready-to-be-delivered message
from the other stream. Maybe this ambiguity only manifests itself with
at least 3 streams involved? Probably another research topic :-)

....
> I am glad that Qiaobing is being rigorous and asking us to consider very
> carefully whether a change in the design is needed, and inspiring us to
> make our arguments very precise.  Qiaobing, don't let us bully you... as
> the film "Twelve Angry Men" illustrates.. just because someone is "on their
> own", doesn't mean they are wrong.   In that film, a jury that starts out
> 11-to-1 in favor of a guilty verdict, eventually reaches a unanimous
> verdict of "not guilty."  We may disagree on "taste" or "philosophy", but
> if we are in disagreement about things that we should be able to logically
> reason out, then we probably need to continue the dialogue until we have
> true consensus.

Thank you, Phill! I am truly really surprised to hear this kind of nice
words, especially on a mailing list like this one in IETF :-)

regards,
-Qiaobing

> Thanks,
> Phill

---

From: Randall Stewart <[email-masked]>
Date: Wed, 05 Jun 2002 11:44:49 -0500

Brian/Qiaobing:

R

---

From: Randall Stewart <[email-masked]>
Date: Wed, 05 Jun 2002 18:43:28 -0500

> that the reciever has more work to do for FORWARD TSN without a forward

Yeah.. I basically had to make a special linked list set that the
stream queues could be placed on.. a bit more efficient but a
reall hassel to maintian a "wheel" (as I call it) of streams with
pending data on it... 

> re-examined whether there are SSN gaps or not.)

Exactly.. I constantly must place stream q;s on and off the :"wheel" no
matter if a fwd-tsn is EVER received...

> list than at the receiver and non-PR streams would never need to be

And it must continue to KEEP this information until it is sure the
peer has received and moved the cum-ack forward..

What I end up doing on the sender side is I hold just the bookkeeping
structures of the TSN (stream seq/stream no/tsn ... etc) but I
drop the mbuf holding the data so I gain a bit of help in mbuf
utilization.

But I can't get rid of the chunk structure UNTIL the cum-ack says he
got my FWD-TSN and accepted it...

So there is NO additional info to save on the sender side and only 
do I save what I must... on the receiver side I have a LOT of extra
bookkeeping.... 

In fact with your confirmation I would claim just the opposite of
Phil's statement... that doing it the way it is NOW documented takes
MORE overhead.. only by adding the SSN/SID can you reduce this..

By the way Renee also pointed out the extra overhead for the IOS
implementation as well :>... so this makes three different
implementations
NOT 2.... 

I plan on adding this to the next pr-sctp unless Michael Tuexen
violently
agrees with Qiaobing ... 

I still await Michaels answer????

---

From: Randall Stewart <[email-masked]>
Date: Wed, 05 Jun 2002 11:46:25 -0500

> be adjusted forward to the SSN contained in the Forward SSNs.

After delivering every stuck SID that could be delivered ... right?

R

---

From: Randall Stewart <[email-masked]>
Date: Wed, 05 Jun 2002 18:35:01 -0500

> sending Fwd-TSN is *deliberately* crafted to be the same as that for

You make a rather larger assumption here Q... i.e. that TSN119 was
lost at the SAME time as the others... this does NOT have to be
the case... it would be if you had a single bundled packet loss.. but
I could also see it be due to multiple packet losses in the network.
This is very very common... and they do NOT have to be at the same
time.... so say the FR algorithm may trigger on the ones up to 118 a
SACK
or so AHEAD of 119's...

I have ALWAYS thought that having the streams info of dropped streams
in the FWD-TSN would make it MUCH easier to implement for the reciever
and not very complex for the sender (since most of the state is
already present anyway)... you know this... but the HOL blocking issue
gives a much more concrete reason to do it...

> same, would the analysis still hold?

of course not.. since then everything is resolved..

R

---

From: Phillip Conrad <[email-masked]>
Date: Wed, 05 Jun 2002 16:37:17 -0400

Dear Colleagues:

Bottom line:
========

Cheers,
Phill

/ptc

---

From: Randall Stewart <[email-masked]>
Date: Wed, 05 Jun 2002 16:57:56 -0500

A few comments..

I agree :>

I agree..

agreed

Sounds good..

R

---

From: Phillip Conrad <[email-masked]>
Date: Thu, 06 Jun 2002 00:15:27 -0400

Replying to several notes at once here... first a reply to some of Randy's 
comments.. then a follow up on a conversation between Qiaobing and Brian.

...

I should have been more precise in my language:  what I intended to convey 
is that the "on-the-wire protocol is simpler"... i.e., fewer fields to 
describe, parse and process.    But as your argument illustrates, when 
considering simplicity/complexity, we need to think of the system as a 
whole, not just one aspect of it.
...

>being lost.. either way is the same result I would think..

Hmmm... I think you and I are saying the same thing here.    The point is 
that 10 chunks might mean ten packets were dropped in a burst, or it might 
mean only one packet was dropped that contained 10 bundled chunks.  From 
the standpoint of chunk processing in SCTP, it does really matter... and 
neither of these is a particularly far fetched scenario.

>things correlate to a particular stream.. but this is just an

Understood... As you point out, the current spec (unlike the
previous U-SCTP spec) does not designate particular streams as reliable or
unreliable; instead indvidual messages are designated this way on the basis
of their lifetime parameters.

What I'm talking about here is just an application scenario
for purposes of illustration.   As part of the argument that  Brian's scenario
is not a corner case, we need to argue that the scenario he describes can
arise is some reasonably normal usage of SCTP.

So for purposes of  illustration only, it is reasonable
to assume  that a given application might "so happen" to use
"k" out of "n" streams for messages with  lifetime parameters
that can  expire, and "n-k" streams for messages that are
fully reliable, not mixing the two.

This makes describing the scenario easier, and
also seems like something that an application might likely do.    For those
that like things to be more "concrete" in a "PS" to this note, I describe a
specific application scenario to motivate this situation.

Now turning to another discussion:

>Please take my little challenges below as a way to help you making your

Qiaobing... thanks for this... I take your comments in that spirit.  And I 
think you've found some weaknesses in my argument that can be tightened up.

>leading up to the loss of the TSN.

I think Brian has it right here.. but I just want to clarify to make sure I 
understand.   Qiaobing is correct in asserting that in the base SCTP 
protocol, it is true that if packets from multiple streams are bundled, the 
loss of a single packet can impact multiple streams.  This is true in the 
base SCTP.

However, there is a property of the base SCTP protocol that can be 
expressed as follows: the loss of a chunk on stream i does NOT delay 
delivery of a chunk on stream j, where i!=j.   Only the loss of lower 
numbered chunks on stream j can affect the delivery of a chunk on stream 
j.   Thus, the base SCTP protocol avoids a particular kind of HOL 
blocking.   (Now this principle is not strictly true even in the base SCTP, 
if one considers the effects of rwnd and cwnd... but at least from the 
standpoint of "ordered delivery", this principle holds.)

Thanks,
Phill

*PS: For those who need a specific application scenario, consider a 
real-time application in manufacturing process control that is sending one 
stream of data that is critical to archive (perhaps emissions reports that 
must be kept for the EPA and thus must be sent reliably) along with some 
other streams of data (temperature, pressure, etc)  where timeliness is 
more crucial than having a complete record of the data...   suppose that 
the received data are being graphed in real time, and if a few intermediate 
points are skipped, the person reading the chart can still make sense of 
the data stream.    One could imagine a similar setup for remote control of 
an robotic vehicle doing toxic waste cleanup, mine sweeping, etc.

---

From: Michael Tuexen <[email-masked]>
Date: Fri, 7 Jun 2002 19:48:52 +0200

Dear all,

reading the thread I would like to make some remarks:

1. I agree that using SSN in the FWD chunk avoids HOL (which is good).
2. Brians example show how HOL can occur if no SSN info in the FWD chunk
    is used. He only looks at the receiver.

Brians example are always based on the assumption that there is a GAP
in the TSN space at the receiver side and the receiver gets a FWD TSN
which does not fill the whole gap. Then the HOL happens at the receiver.

I'm just wondering how often this will happen. What are the reasons that
a FWD chunk will be send: RTX Timer expiration or fast retransmission.
In both cases I would send a FWD chunk and bundle the remaining DATA 
chunks
(if possible). So there is a good chance that the whole gap is filled by
the FWD chunk and the retransmitted DATA chunks.
I'm not sure how likely Brians scenario is taking the sender into 
account.

So I think one has to take the senders behavior into account when 
discussing
this issue.

Best regards
Michael

>>>>> Michael T/Phil? what do you think... Michael R.. I will talk to you 
[email-masked]

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Fri, 7 Jun 2002 15:39:11 -0500

Michael, Qiaobing,

Here is the sender side of the example.  It is quite reasonable
that this could occur.  The receiver is blocked on streams 1
through 9 until retransmission of TSN 119 is successful.  This
is the sender side of the same example.

    TSN 101  SID 1   SSN 1   |
    TSN 102  SID 2   SSN 1   |
    TSN 103  SID 3   SSN 1   |
    TSN 104  SID 4   SSN 1   |   SCTP Packet
    TSN 105  SID 5   SSN 1   +--------------------->
    TSN 106  SID 6   SSN 1   |
    TSN 107  SID 7   SSN 1   |
    TSN 108  SID 8   SSN 1   |
    TSN 109  SID 9   SSN 1   |

    TSN 110  SID 1   SSN 2   |
    TSN 111  SID 2   SSN 2   |
    TSN 112  SID 3   SSN 2   |
    TSN 113  SID 4   SSN 2   |   SCTP Packet
    TSN 114  SID 5   SSN 2   +----------X
    TSN 115  SID 6   SSN 2   |         Lost
    TSN 116  SID 7   SSN 2   |
    TSN 117  SID 8   SSN 2   |
    TSN 118  SID 9   SSN 2   |
                                 SCTP Packet
    TSN 119  SID 0   SSN X   |----------X
                                       Lost

    (Timed reliability expiry on TSNs 110 to 118)

    FORWARD TSN 118          |--------------------->

    TSN 120  SID 1   SSN 3   |
    TSN 121  SID 2   SSN 3   |
    TSN 122  SID 3   SSN 3   |
    TSN 123  SID 4   SSN 3   |   SCTP Packet
    TSN 124  SID 5   SSN 3   +--------------------->
    TSN 125  SID 6   SSN 3   |
    TSN 126  SID 7   SSN 3   |
    TSN 127  SID 8   SSN 3   |
    TSN 128  SID 9   SSN 3   |

    (Receiver is now blocked on streams 1 through 9.)

    (Some time later, retrans timeout on TSN 119)

                                 SCTP Packet
    TSN 119  SID 0   SSN X   |--------------------->

Q.E.D.

--brian

On Fri, 07 Jun 2002, Qiaobing Xie wrote:

> Michael,
> 
> Right on! You summarized my previous point beautifully. I start to
> realize that lately my arguments tend to become lengthy and off the
> point... maybe a sign of getting old :<
> 
> So far I kind of buy Randy's point better that "adding some bits in
> exchange for ease of implementation", but still yet to be convinced
> about the HOL blocking argument made by Brian. 
> 
> I've asked for an "all-considered" example of the HOL blocking, but
> haven't seen one yet.
> 
> regards,
> -Qiaobing
> 
> Michael Tuexen wrote:
> > 
> > Dear all,
> >

---

From: Qiaobing Xie <[email-masked]>
Date: Fri, 07 Jun 2002 14:49:50 -0500

Michael,

regards,
-Qiaobing

---

From: Randall Stewart <[email-masked]>
Date: Fri, 07 Jun 2002 11:37:08 -0500

Q:

> Sadly but that seems to be the case :-)

I must agree here :<  

Having gone through this debate before with you in private
I feel rather badly that I have "changed my mind" in public :0
But as you know I always thought it was more efficent processing 
wise but I did agree that if there were no other benefit to the
bits... then this would just be a "cost to pay". But I think
with Brian's additional HOL issue it changes the balance.

I do think this thread has had a very calm tone (compared to
issues in the past)... I don't think anything was said in anything
but concern for the BEST solution that we all can reach "rough"
consensus
on :>

> the same as what's called "designed by committed". We are better than

Here I could not agree with you more :>

Q: When I get back from my travels I will see if I can't make ammends
   to you on my "change" by offering you a glass of cab at my house :>

R

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Fri, 7 Jun 2002 19:49:54 -0500

Qiaobing,

> >     TSN 105  SID 5   SSN 1   +--------------------->

        <---- SACK CUM ACK 109

> >     TSN 115  SID 6   SSN 2   |         Lost

        <---- SACK CUM ACK 109

> >     (Timed reliability expiry on TSNs 110 to 118)

        <---- SACK CUM ACK 109

> Why TSN119 is not re-sent at this exactly same moment? As Michael and I
> both noted that, in PR-SCTP, Fwd TSN can only be triggered due to either
> a "RTX Timer expiration or fast retransmission." Could you fill in the
> details that lead to TSN119 not being re-sent at this moment?

With the SACKs shown, the FORWARD TSN "MUST" be sent according to
Rule C3.  Retransmission of TSN 119 has nothing to do with it.

--brian

---

From: Qiaobing Xie <[email-masked]>
Date: Fri, 07 Jun 2002 19:39:17 -0500

---

From: Qiaobing Xie <[email-masked]>
Date: Fri, 07 Jun 2002 19:41:28 -0500

...
>    to you on my "change" by offering you a glass of cab at my house :>

This is a nice offer and I will take it :-)

-Q

---

From: Michael Tuexen <[email-masked]>
Date: Sat, 8 Jun 2002 13:48:38 +0200

Brian,

see my questions below.

Best regards
Michael

>>>     TSN 115  SID 6   SSN 2   |         Lost
Why is this SACK be send? There is no event at the receiver since the 
packet is lost on the wire.
>>>     (Timed reliability expiry on TSNs 110 to 118)
Why is this SACK be send
>>>     FORWARD TSN 118          |--------------------->
This now means that the retransmission timer runs off. So bundle a 
FWD-TSN and DATA.
So I really do not understand this example.
>>>>>>>>>       I agree with Randy that Brian has raised an important 
>>>>>>>>> issue
>>>>>>>>> for us to
>>>>>>>> I must disagree with your "it is simpler" ... it is NOT. The 
>>>>>>>> question
>>>>>>>> is
>>>>>>>>> that the situation Brian describes is a "corner case" or not.   
>>>>>>>>> So
>>>>>>>>> I've
>>>>>>>>> is a corner case at all.   So, I'm inclined to agree with Randy 
>>>>>>>>> and
>>>>>>>>> Brian
>>>>>>>>> But I'd like to invite others to walk through my line of 
>>>>>>>>> reasoning
>>>>>>>>> to test
>>>>>>>>> In Brian's example, messages are being sent round-robin on a 
>>>>>>>>> number
>>>>>>>>> of
>>>>>>>>> streams.  This seems like a fairly reasonable use of SCTP 
>>>>>>>>> streams,
>>>>>>>>> do that
>>>>>>>>> OK.   This does not strike me as particularly unusual; a loss 
>>>>>>>>> of a
>>>>>>>>> single
>>>>>>>>> packet followed by a successful packet is sufficient to cause 
>>>>>>>>> this
>>>>>>>>> situation.
>>>>>>>>>
>>>>>>>>
>>>>>>>>> TSN.   However, no Forward TSN is issued for TSN119.  This may 
>>>>>>>>> be
>>>>>>>>> because
>>>>>>>>> the messages on stream 9 are treated as Reliable.   This is NOT 
>>>>>>>>> an
>>>>>>>>> unusual
>>>>>>>>> situation; it is perfectly reasonably to assume that an 
>>>>>>>>> application
>>>>>>>>> might
>>>>>>>>> have "k" streams with data that can be dropped, and "n - k" 
>>>>>>>>> streams
>>>>>>>>> of data
>>>>>>>>> that must be treated as reliable.  In the case of Brian's 
>>>>>>>>> example, n
>>>>>>>>> is 10,
>>>>>>>>> and k is 9, but I think that his example will still produce 
>>>>>>>>> similar
>>>>>>>>> results
>>>>>>>>> potentially being HOL blocked by loss of a single packet.  I 
>>>>>>>>> think
>>>>>>>>> we need
>>>>>>>>> to fix this.
>>>>>>>>
>>>>>>>> I agree..
>>>>>>>>
>>>>>>>>>
>>>>>>>>> The solution of forward SSNs is one viable solution I think.  
>>>>>>>>> There
>>>>>>>>> may be
>>>>>>>>> others, but that's as good as any for right now I think.   
>>>>>>>>> There may
>>>>>>>>> be
>>>>>>>>> another solution as well, but let me do some writing and 
>>>>>>>>> thinking
>>>>>>>>> about it
>>>>>>>>> off line before I muddy the waters with it here... just to make 
>>>>>>>>> sure
>>>>>>>>> it is
>>>>>>>>> a viable alternative.
>>>>>>>>
>>>>>>>> agreed
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Randy... I'm willing to make the text changes and send a 
>>>>>>>>> proposed
>>>>>>>>> version
>>>>>>>>> around to the co-authors for comment if there is consensus to 
>>>>>>>>> move
>>>>>>>>> in that
>>>>>>>>> direction.   Let me know.
>>>>>>>>>
>>>>>>>>
>>>>>>>>>> After watching this debate.. I think I have come to the 
>>>>>>>>>> conclusion
>>>>>>>>>> that
>>>>>>>>>> Brian is correct :)
>>>>>>>>>>
>>>>>>>>>> NOT mean 4 bytes per TSN but 4 bytes per where one or more 
>>>>>>>>>> TSN's
>>>>>>>>>> were
>>>>>>>>>> dropped..
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Michael T/Phil? what do you think... Michael R.. I will talk 
>>>>>>>>>> to you
>>>>>>>>>> at
>>>>>>>>>> the
>>>>>>>>>> next class break..
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> R
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Randall R. Stewart

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Sat, 8 Jun 2002 12:16:37 -0500

Michael,

> >>>     (Timed reliability expiry on TSNs 110 to 118)
To acknowledge the first SCTP Packet

> This now means that the retransmission timer runs off. So bundle a 

No it does not: according to rule C3 a SACK has been received.  The
retrans timer has not expired. We are forced to send FORWARD TSN
just because a sack has arrived.

> So I really do not understand this example.

You have no imagination: remove the first two SACKs.
Then it is completely possible.

--brian

---

From: Michael Tuexen <[email-masked]>
Date: Sat, 8 Jun 2002 19:59:00 +0200

Brian, Qiaobing,

I'm a bit confused by the answer, because there are some SACKs less than
in the previous email.

I will repeat the scenario:

Consider
packet1 = TSN 101-109; SID=1-9; SSN=1;
packet2 = TSN 110-119; SID=1-9; SSN=2;
packet3 = TSN 119;     SID=0;
packet4 = TSN 120-128; SID=1-9; SSN=3;

It seems that you are now considering the szenario:
1. packet1 is transmitted and received
2. packet2 is transmitted and lost on the wire
3. packet3 is transmitted and lost on the wire
4. timed reliability runs off for TSN 110-119
5. packet4 is transmitted and received. The user data can not be 
delivered.
6. A SACK for packet 1 is received.
7. A FWD TSN chunk is sent

This happens only if the timed reliability is shorter than the round 
trip time.
But we can assume this.

So what will happen:
8. The retransmission timer will run off for packet3.
9. packet3 gets retransmitted, and the user data of packet3 and packet4 
can be delivered.

I agree with you that there is HOL, but it is the same as there would be 
if you
would not use PR-SCTP. So PR-SCTP, as it is currently being defined does 
not perform
better than basic SCTP in terms of user-to-user message delay. It does 
perform better in terms
of buffer and bandwidth usage.

So I think using SSN info in the FWD TSN chunk would optimize PR-SCTP 
also in terms of
user-to-user delay. Qiaobing, could you agree on this?

Best regards
Michael
On Saturday, June 8, 2002, at 07:16 PM, Brian F. G. Bidulock wrote:

>>>>>>> I'm just wondering how often this will happen. What are the 
>>>>>>> reasons
>>>>>>> that
>>>>>>>>>> where is the burden. Having implemented pr-sctp in the kernel 
>>>>>>>>>> of
>>>>>>>>>> freebsd
>>>>>>>>> FORWARD TSN.  This is because it is not possible for the 
>>>>>>>>> receiver
>>>>>>>>> to
>>>>>>>>> the load on the receiver is increased even when this case does 
>>>>>>>>> not
>>>>>>>>> occur.
>>>>>>>>> (Each time gaps close, the next expected SSN for all streams 
>>>>>>>>> need
>>>>>>>>> to be
>>>>>>>>> assign the next SSN (++SSN).  Indeed the sender when 
>>>>>>>>> generating a
>>>>>>>>> FORWARD
>>>>>>>>>>> The question of whether to choose A or B depends on whether 
>>>>>>>>>>> you
>>>>>>>>>>> believe
>>>>>>>>>>> walked through Brian's example, and have determined that I 
>>>>>>>>>>> don't
>>>>>>>>>>> think it
>>>>>>>>>>> Now for the next part of this explanation, I'm going to have 
>>>>>>>>>>> to
>>>>>>>>>>> infer
>>>>>>>>>>> certain things, so, Brian, please correct me if I infer 
>>>>>>>>>>> wrongly.
>>>>>>>>>>> *One*
>>>>>>>>>>> situation Brian describes is that there is a burst loss of 
>>>>>>>>>>> data
>>>>>>>>>>> chunks
>>>>>>>>>>> bundled!)... and then data chunks TSN120 through TSN128 make 
>>>>>>>>>>> it
>>>>>>>>>>> through
>>>>>>>>>> Or the other possiblility would be a set of bundled chunks in 
>>>>>>>>>> one
>>>>>>>>>> packet
>>>>>>>>>> set to "fully reliable". One possilbe implementation would be 
>>>>>>>>>> to
>>>>>>>>>> make
>>>>>>>>>>> decides to no longer transmit and/or retransmit a specific 
>>>>>>>>>>> data
>>>>>>>>>>> chunk.  How
>>>>>>>>>>>> I think it is reasonable and not that much overhead to 
>>>>>>>>>>>> include
>>>>>>>>>>>> JUST
>>>>>>>>>>>> the dropped streams NEXT expected SSN... You would be 
>>>>>>>>>>>> adding 4
>>>>>>>>>>>> bytes
>>>>>>>>>>>> I think this is reasonable.. it improves processing AND 
>>>>>>>>>>>> avoids
>>>>>>>>>>>> HOL
>>>>>>>>>>>> blocking..
>>>>>>>>>>>>
>>>>>>>>>>>>

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Sat, 8 Jun 2002 14:26:12 -0500

Michael,

> packet2 = TSN 110-119; SID=1-9; SSN=2;

I think you mean TSN 110-118 here.

> This happens only if the timed reliability is shorter than the round 

No that is not necessary.  The SACK (containing no gaps) may be
a "delayed" sack delayed by 500ms so timed reliability only needs to
be shorter than the current (experienced by the SACK) RTT + 500ms.
Nevertheless, it is easy for a fixed "timed reliability" to be less
than the current RTT+500ms at any given point, particularly when
packets are being lost in the network due to congestion.

> But we can assume this.

The retransmission timer may be as high as 60 seconds at the moment,
so it is quite conceivable that "timed reliability" is less than the
sender's RTO.  In fact, for "timed reliability" to work as it is
expected, the timed reliability interval MUST be less than the product
of Assoc.Max.Retrans and Rto.Max, otherwise, packets will never expire.
(The association will fail before they expire.)

> of buffer and bandwidth usage.

I disagree with that analysis.  Normal SCTP only has to wait for
retransmission of the SSN 2 packet, whereas PR-SCTP in this scenario
has to wait for retransmission of all the TSN gaps subsequent to
the FORWARD TSN.  These (e.g. TSN 119) might have 16k worth of data
in it making it subject to many retransmissions (the network is already
marginally in congestion).  The SSN 2 packet might contain 256 bytes
meaning that retransmission of TSNs 110-118 could be successful when
TSN 119 is going to be unsuccessful for some time.

It is NOT the situation in regular SCTP that a stream must block
awaiting the retransmission of a TSN from a different stream.  For
PR-SCTP, streams 1 through 9 are blocked awaiting the retransmission
of a data chunk on stream 0.  This is the unacceptable HOL blocking.
It is inter-stream, not intra-stream.

--brian

---

From: Michael Tuexen <[email-masked]>
Date: Sat, 8 Jun 2002 22:31:33 +0200

Brian,

see my comments below.

Best regards
Michael

> I think you mean TSN 110-118 here.
You are correct, it is a typo.
> packets are being lost in the network due to congestion.
Yes, the SACK can be delayed.
> It is inter-stream, not intra-stream.
Hmmm. I see. The user data of packet4 can be delivered to the upper layer
after the FWD-TSN (something like a retransmission of packet2) and 
packet3
have been received. In normal SCTP only packet2 needs to be received.
Since the user-to-user delay of PR-SCTP should not be greater than for
SCTP, I think we should put the SSN/SID info in the FWD-TSN chunk.
But only for the affected streams. Qiaobing, what do you think?
Best regards
Michael
>>>>>>>>> 2. Brians example show how HOL can occur if no SSN info in the 
>>>>>>>>> FWD
>>>>>>>>> chunk
>>>>>>>>> Brians example are always based on the assumption that there 
>>>>>>>>> is a
>>>>>>>>> GAP
>>>>>>>>> in the TSN space at the receiver side and the receiver gets a 
>>>>>>>>> FWD
>>>>>>>>> TSN
>>>>>>>>>>>>> consider, and I think has explained it quite well with a 
>>>>>>>>>>>>> very
>>>>>>>>>>>>> clear,
>>>>>>>>>>>>> illustrative example.
>>>>>>>>>>>>>
>>>>>>>>>>>> an extra set of linked lists and such to track any stream 
>>>>>>>>>>>> that
>>>>>>>>>>>> has
>>>>>>>>>>> I agree.  I discovered this situation while implementing 
>>>>>>>>>>> PR-SCTP
>>>>>>>>>>> in the
>>>>>>>>>>> Linux Kernel.  The normal receiver can rely on the fact that 
>>>>>>>>>>> the
>>>>>>>>>>> next
>>>>>>>>>>> a forward SSN causes the receive to have to update this 
>>>>>>>>>>> expected
>>>>>>>>>>> value
>>>>>>>>>>> with the next SSN in the reorder queue whenever there are no 
>>>>>>>>>>> TSN
>>>>>>>>>>> gaps.
>>>>>>>>>>> that the reciever has more work to do for FORWARD TSN 
>>>>>>>>>>> without a
>>>>>>>>>>> forward
>>>>>>>>>>> SSN.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> know a priori which streams could be affected by this 
>>>>>>>>>>> scenairo.
>>>>>>>>>>> Therefore,
>>>>>>>>>>> It is the mere possibility of its occurence which introduces 
>>>>>>>>>>> the
>>>>>>>>>>> load.
>>>>>>>>>>> TSN chunk would need to traverse outbound stream structures 
>>>>>>>>>>> with
>>>>>>>>>>> dropped
>>>>>>>>>>> list than at the receiver and non-PR streams would never need 
>>>>>>>>>>> to
>>>>>>>>>>> be
>>>>>>>>>>> involved.
>>>>>>>>>>
>>>>>>>>>> But I can't get rid of the chunk structure UNTIL the cum-ack 
>>>>>>>>>> says
>>>>>>>>>> he
>>>>>>>>>>>>> chunks, but avoids re-introducing an HOL blocking problem, 
>>>>>>>>>>>>> and
>>>>>>>>>>>>> thus
>>>>>>>>>>>>> TSN110 through TSN119 (ten chunks, but possibly less than 
>>>>>>>>>>>>> ten
>>>>>>>>>>>>> packets..
>>>>>>>>>>>>> perhaps even only ONE packet if the chunks are small enough 
>>>>>>>>>>>>> to
>>>>>>>>>>>>> be
>>>>>>>>>>>>> the technical sense of that word; see the defn in the 
>>>>>>>>>>>>> current
>>>>>>>>>>>>> draft.  This
>>>>>>>>>>>> things that I have thought about that must be covered... I 
>>>>>>>>>>>> had
>>>>>>>>>>>> something
>>>>>>>>>>>>> "dropped" would cause confusion, and it already has.   
>>>>>>>>>>>>> Perhaps
>>>>>>>>>>>>> should use
>>>>>>>>>>>>> another word in the draft to describe the operation where 
>>>>>>>>>>>>> the
>>>>>>>>>>>>> sender
>>>>>>>>>>>>>> considered U-SCTP. Renee Revis first pointed this out to 
>>>>>>>>>>>>>> me..
>>>>>>>>>>>>>> but
>>>>>>>>>>>>>> in the FWD TSN... I did NOT think of the HOL blocking 
>>>>>>>>>>>>>> issue..
>>>>>>>>>>>>>> or
>>>>>>>>>>>>>> Michael R. is here... I will discuss it with him.. I want 
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>> get
>>>>>>>>>>>>>> the other co-author's take on this one... If the others 
>>>>>>>>>>>>>> agree
>>>>>>>>>>>>>> with
>>>>>>>>>>>>>> Brian (and me :>) then I think we will put this in.... 
>>>>>>>>>>>>>> also I
>>>>>>>>>>>>>> would
>>>>>>>>>>>>>> for EVERY stream dropping one or more TSN's... note that 
>>>>>>>>>>>>>> this
>>>>>>>>>>>>>> does

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Sat, 8 Jun 2002 15:56:26 -0500

Michael,

Good.  Now if we can convince Qiaobing it will be unanimous.

--brian

[snip]
> But only for the affected streams. Qiaobing, what do you think?
[snip]

---

From: Qiaobing Xie <[email-masked]>
Date: Mon, 10 Jun 2002 12:44:21 -0500

Brian,

As I said before, I have no strong objection to the change. Still,
Randy's adding SNN info to make the implementation easier sounds a much
better argument to me. 

The reason that I encouraged you to fill in the precise details to your
initial example should become apparent now. As demonstrated by the
exercise you and Michael went through, to show that the absence of SNN
info in Fwd TSN can really cause a meaningful delay at the receiver end
is not as easy as you might originally think. That is my point - yes, it
can cause delay but you need to assume quite a few things happening
simultaneously for the delay to become meaningful. 

-Qiaobing