Subject: Adopting draft-tuexen-tsvwg-sctp-sack-immediately as WG item? - 29-Aug-2011

From: Gorry Fairhurst <[email-masked]>
Date: Tue, 16 Aug 2011 10:06:43 +0100

The authors of "SACK-IMMEDIATELY extension for the Stream Control
Transmission Protocol draft-tuexen-tsvwg-sctp-sack-immediately-06.txt"
requests that this is adopted as WG item.

    This document defines a method for a sender of a DATA chunk to
    indicate that the corresponding SACK chunk should be sent back
    immediately.

There was hum indicating general WG support for this at the last IETF in 
  Quebec City, and the document appears mature to complete quickly. This 
  email is to allow people to confirm that there is sufficient energy to 
  complete this work in TSVWG.

* PLEASE send an email to this list if you think this work needs to be
done in the IETF (even if you previously noted this at the IETF).

* PLEASE send an email to this list if you think adopting this document
in TSVWG is a good *OR* a bad idea.

* Please also indicate if you are willing to REVIEW such a document
during its development - our AD has indicated that we expect to have at
least 4 people who commit to review WG outputs. If you can promise to do 
this please say!

Please provide any comments at the latest by 29th August 2011.

Best regards

Gorry & James
(TSVWG Co-Chairs)

---

From: Gorry Fairhurst <[email-masked]>
Date: Tue, 09 Aug 2011 08:36:10 +0100

Please provide any comments at the latest by 23rd August 2011.

Best regards

Gorry & James
(TSVWG Co-Chairs)

---

From: Thomas Dreibholz <[email-masked]>
Date: Tue, 16 Aug 2011 21:30:22 +0200

Dear Gorry,

I have read the document and think it is a very useful extension to SCTP. I 
also work with the SACK Immediately implementation within the OMNeT++ INET 
framework.

Of course, I am also willing to review this document during further 
development. 

Best regards
-- 
=======================================================================
 Dr. Thomas Dreibholz

 University of Duisburg-Essen,                   Room ES210
 Inst. for Experimental Mathematics              Ellernstraße 29
 Computer Networking Technology Group            D-45326 Essen/Germany
-----------------------------------------------------------------------
 E-Mail:     [email-masked]
 Homepage:   http://www.iem.uni-due.de/~dreibh
=======================================================================

---

From: "Becke, Martin" <[email-masked]>
Date: Wed, 17 Aug 2011 07:45:46 +0000

> * PLEASE send an email to this list if you think adopting this document in
>  TSVWG is a good *OR* a bad idea.

I think it is a good idea and I'm very interested in this draft.

> * Please also indicate if you are willing to REVIEW such a document during its
> development

I can be a reviewer.

Best regards

Martin Becke

 Martin Becke

 University of Duisburg-Essen,                   Room 308a
 Inst. for Experimental Mathematics              Ellernstraße 29
 Computer Networking Technology Group            D-45326 Essen/Germany
-----------------------------------------------------------------------
 Phone :     +49-201-183-7667
 E-Mail:     [email-masked]
=======================================================================

---

From: Mark Allman <[email-masked]>
Date: Wed, 17 Aug 2011 08:04:41 -0400

> Transmission Protocol
> draft-tuexen-tsvwg-sctp-sack-immediately-06.txt" requests that this
> is adopted as WG item.
> 

Well, I don't quite understand the motivation.  I read the draft quickly
and still I don't quite get it.

The reasons given are ...

>  o  The sender has not enough queued user data to send the remaining
>     DATA chunks due to the Nagle algorithm.

OK.  But, why then is it important to get a quick ACK?  It isn't like
the sender is waiting to do anything.

>   o  The sending of a DATA chunk fills the congestion or receiver
>      window.

Again, why is an immediate ACK important here?  In steady state if you,
say, fill the congestion window then packets on the left side of the
window will free up space in the window well before some tagged packet
all the way on the right side of the window is processed.

>   o  The sender is in the SHUTDOWN-PENDING state.

I don't recall my SCTP state diagram very well.  But, can't the receiver
detect this and switch to an immediately ACKing strategy at that point?

>   o  The sender has reduced its RTO.Min such that a retransmission
>      timeout will occur if the receiver delays its SACK.

This is self-fulfilling.  The RTO *should* include the delayed ACK
time.  The RTO is not an RTT of some sort, but rather the time until
something should be *retransmitted*.  So, the samples you get will be
naturally taking into account any delay in the ACKing process and the
RTO should then accommodate that extra delay.

>   o  The application requests to set the I-bit of the last DATA chunk
>      of a user message when providing the user message to the SCTP
>      implementation.

Again, why does this really matter?  You can't get a message from A to B
faster if you flip a magic bit.  And, it isn't clear to me that the ACK
From B to A really matters that much to apps or users.  Surely it has to
come, but I don't think we're generally sitting on the edge of our seat
awaiting it, right?  If there is data to be sent by B in response then
that'll get sent by B according to the rules and an ACK will get
piggybacked.

So, basically:

  (1) I don't think this proposal for a bit is particularly harmful or
      burdensome or will cause big problems in some way.

  (2) I don't understand the need and it seems the draft could at least
      do a much better job explaining why this functionality is
      desirable (if in fact it is).

FWIW,
allman

---

From: Michael Tüxen <[email-masked]>
Date: Wed, 17 Aug 2011 21:29:49 +0200

> the sender is waiting to do anything.
Well, it depends. One particular example which comes up to my mind is
a simple request/response protocol running on top of SCTP.
Without pipelining the request (which is a good thing) you would
expect a rate of 1 response per RTT. What people run into is the
following. They test with small request size (small means that
it fits into an MTU) and get the expected behavior. When the request
size is larger than an MTU but smaller than two MTUs they get
5 responses per second. Why? Because the SCTP stack sends the first
fragment to the peer, but delays sending the second fragment since
there is outstanding data due to the Nagle algorithm. The peer
does delay the SACK for 200ms. After sending the SACK, the second
fragment of the request gets sent and the response comes back.
The use can avoid this by:
1. Disabling Nagle at all.
2. Pipelining multiple requests.
The first is doable in any case, the second depends on the application.
However, the problem shows up only if the number of packets for the
request is even. Based on questions I got in the past, it is not
clear for the app writer to deal with these problems. So letting the
stack doing it seems useful.
We could also disable Nagle by default, but this is a bad idea I think.
> all the way on the right side of the window is processed.
We ran some tests where we had two competing SCTP associations. One
sending a SACK for every packet, one sending it for every other with
a delay ACK timer. We varied the timer.
The result was that the bandwidth was not shared equally. The
association sending a SACK for every packet got the higher
throughput. Using the I-Bit as described above mitigates this.
You can find more details in a paper with the title:
Improving the Acknowledgment Handling of SCTP
> detect this and switch to an immediately ACKing strategy at that point?
No, the receiver doesn't know.
When the sender calls close() the sender waits until there is no outstanding
data anymore and then sends a SHUTDOWN. So the sender has to wait for the
SACK and the receiver has no idea.
> RTO should then accommodate that extra delay.
I agree with you, it is a misconfiguration here. But I have seen
this being deployed... The sender can detect this but can't do
anything. Using the I-Bit it has a chance...
> that'll get sent by B according to the rules and an ACK will get
The point you are missing here is that there are cases in which the
sender delays a message until is has no outstanding data anymore.
This happens, for example, in the DTLS/SCTP handshake. SCTP even
provides an API for getting notified when there is no outstanding
data anymore.
I consider DTLS/SCTP the best way of using transport layer security
over SCTP (see http://tools.ietf.org/html/rfc6083 ). In the OpenSSL
implementation we make use of the I-Bit and this reduces the
time to perform the initial handshake by 1 second (5 SACK timeouts).
Without the delay it takes about 3 RTTs. So saving 1 second is
a huge gain.
>      burdensome or will cause big problems in some way.
I agree. In the worst case you get the same behavior as the receive
having switched off the delaying of SACKs. So it does't hurt, but
you gain something. How much, depends on the particular scenario.
>      desirable (if in fact it is).
Maybe I was able to clarify it a bit. It it is clearer for you, I can
try to integrate some of the above into the ID to make it clearer.

Thanks you very much for your comments.

Best regards
Michael

---

From: Janardhan Iyengar <[email-masked]>
Date: Sat, 20 Aug 2011 18:45:04 -0400

Gorry, James,

> least 4 people who commit to review WG outputs. If you can promise to do this please say!

As promised in Quebec City, I will read and review the document soon.

Thanks,
- jana

-- 
Janardhan Iyengar
Assistant Professor, Computer Science
Franklin & Marshall College
http://www.fandm.edu/jiyengar

---

From: Kacheong Poon <[email-masked]>
Date: Tue, 23 Aug 2011 00:52:39 +0800

> We could also disable Nagle by default, but this is a bad idea I think.

The above scenario was discussed in the context of TCP
many years ago.  And AFAIK, most current TCP stacks do
not have this issue anymore because the aforementioned
way to implement Nagle algorithm has been determined to
be the wrong way.  There is now no delay in sending the
second TCP segment in the above scenario.  There is no
need to modify the TCP protocol.  It is just an
implementation issue.  If there is no issue in TCP, I
guess there should also be no issue in SCTP.

In fact, it is much easier to fix the SCTP implementation
than to deploy a protocol level change.  The former
only requires the sender SCTP stack to be fixed while the
latter requires both sides to support it plus app level
changes.

Is a protocol level change really needed?

-- 

					K. Poon.
					[email-masked]

---

From: Michael Tüxen <[email-masked]>
Date: Mon, 22 Aug 2011 19:29:57 +0200

> guess there should also be no issue in SCTP.
So you are arguing that you can change the implementation
of the Nagle algorithm. That is correct. However, that
does not help in the other scenarios:
1. You need to delay the sending of a SHUTDOWN chunk until
   you received a SACK, which might be delayed.
2. You need to wait for a possibly delayed SACK timer if
   you ware waiting for a SENDER_DRY event.
3. You have no chance to deal with misconfigured peers
   in case the RTO values have been lowered on your side,
   but the SACK delay on the peer not. Unfortunately,
   I have seen this multiple times in production networks.
> than to deploy a protocol level change.  The former
As said above: This only helps if the Nagle algorithm is
involved.
BTW: Is there a description (ID/RFC) available on the
Nagle algorithm you describe? At least on FreeBSD we
don't use that, but the old one.
> Is a protocol level change really needed?
"Needed" is the crucial point here.
Needed in the sense that the protocol is broken without
the change: No.
Needed in the sense the the protocol shows a good performance
in some situations: Yes.
In particular taking the complexity of the change involved:
1. For the receiver: trivial.
2. For the sender: very easy.

Best regards
Michael

---

From: Kacheong Poon <[email-masked]>
Date: Tue, 23 Aug 2011 15:40:47 +0800

>     you received a SACK, which might be delayed.

What is the rationale of "optimizing" the closing of an
association?  An app should not need to be involved in
the shutdown sequence.  For example, an app just call
close() and can forget about the association.

>     you ware waiting for a SENDER_DRY event.

Can you explain why this is actually important?

>     I have seen this multiple times in production networks.

As Mark has pointed out, this is not a reason.  In fact,
if the peer has an implementation or configuration
problem, how can you be sure that the peer implements the
quick ACK correctly?  Adding more features won't help fix
misconfiguration or implementation error.  I'd argue that
adding more features actually make things worse as things
are getting even more complicated.  Quick ACK is going to
hurt instead of helping at all.

> As said above: This only helps if the Nagle algorithm is

But it seems to be the single reasonable use of this
mechanism given so far.

> don't use that, but the old one.

I think you can search the tcpm or the old tcp-impl
mailing list.  IMHO, it is just a simple implementation
issue which does not really need an ID...

> Needed in the sense the the protocol shows a good performance

The performance issue mentioned can be solved by fixing
the implementation.  It is not a protocol level issue.

> 2. For the sender: very easy.

No, a transport protocol level change is never a trivial
thing.  IMHO, this is a very dangerous thinking.  Anyway,
just my $0.02.

-- 

					K. Poon.

---

From: Michael Tuexen <[email-masked]>
Date: Tue, 23 Aug 2011 12:12:10 +0200

> close() and can forget about the association.
That is the client side. Think about the server side.
This 200ms delay will require the server to keep the
assoc around for additional 200ms for just nothing.
If you have a server with lots of short living associations,
you might want to get rid of them as soon as possible.
> Can you explain why this is actually important?
If an applications wants to make sure that everything is
received by the peer SCTP stack, it will enable the
subscribe to the sender dry event and wait for it.
If the peer is delaying the SACK for 200ms, it has to
wait RTT + 200ms instead of just the RTT. For small
RTTs this is a difference.
The is used for example in the DTLS handshake. The delays
can add up to 1 second.
> hurt instead of helping at all.
Hmmm. At least in the scenarios I was able to look
at trace files, the receiver was running Linux, which
supports the I-Bit. So getting it implemented was
not a problem (FreeBSD also supports it, Solaris don't,
I guess).
I agree, fixing the configuration problem is the solution.
But in the traces contained the following pattern:
Transmit data, retransmit it, get a SACK with a dup report.
Over and over. So the sender can detect that there is
some problem, could set the I-bit and at least reduce the
traffic on the network. It can also notify the user that
there is a problem.
> issue which does not really need an ID...
... OK. So simple that it was missed in the SCTP implementation
of BSD, I guess the same for Linux, and assume that Solaris does
it right...
Just to get it clear:
You don't delay a DATA chunk if only the E-bit is set, right?
> the implementation.  It is not a protocol level issue.
I don't see this for the server side problem, for the
sender dry problem.
> thing.  IMHO, this is a very dangerous thinking.  Anyway,
I did the implementation in FreeBSD and saw the diff for Linux.
It is simple.
I agree with you, that one has to consider what a user can
do with it if it is not used as intended. But you can't be
worse than you can already be now by sending back a SACK
for each packet...

Best regards
Michael

---

From: Mark Allman <[email-masked]>
Date: Wed, 24 Aug 2011 12:52:07 -0400

> > OK.  But, why then is it important to get a quick ACK?  It isn't
> > like the sender is waiting to do anything.
>
> a simple request/response protocol running on top of SCTP.  Without
> pipelining the request (which is a good thing) you would expect a rate
> of 1 response per RTT. What people run into is the following. They
> test with small request size (small means that it fits into an MTU)
> and get the expected behavior. When the request size is larger than an
> MTU but smaller than two MTUs they get 5 responses per second. Why?
>
> [...]
> 
> The first is doable in any case, the second depends on the
> application.  However, the problem shows up only if the number of
> packets for the request is even. Based on questions I got in the past,
> it is not clear for the app writer to deal with these problems. So
> letting the stack doing it seems useful.  We could also disable Nagle
> by default, but this is a bad idea I think.

I understand the case... we write 1.5 packets and one gets sent and the
other sits there waiting until the first is ACKed (in the hopes that
more data will be sent and it can all be aggregated into a larger
packet).  And, the ACK is further delayed by delayed ACKs at the
receiver.  And, so if you could tell the receiver to ACK immediately
then that'd save a little time.

OK, but that nips at the edge of the problem, right?  First and foremost
this is an issue of the sender not the acknowledger.  I.e., if you have
the cwnd why delay the 0.5 packets?  Sure, we want to aggregate packets
to the extent possible.  But, if that notion is getting in the way for
some app then lets not do it.  Lets either turn Nagle off or let one
short write be outstanding or something.  

It doesn't seem like an "ACK now" bit is really addressing this problem.

Put a different way, if the situation is this ....

  (1) send a full packet, queue half-a-packet
  (2) full packet arrives 0.5*RTT later
  (3) receiver sits around waiting for another packet such that it can
      combine ACKs for some time X
  (4) after X seconds have elapsed the receiver gives up and ACKs, that
      ACK arrives at the sender 0.5*RTT later
  (6) sender gets ACK, sends the half-a-packet
  (7) the half-a-packet arrives 0.5*RTT later
  (8) receiver awaits another packet to coalesce ACKs, gives up after Y
      seconds and sends an ACK
  (9) ACK arrives 0.5*RTT later

So, that whole thing takes 2*RTT + X + Y.

With the proposal you are setting X=Y=0.  The process still takes 2
RTTs.  If you just write the 1.5 packets in step (1) the whole thing
will take 1 RTT.  And, doing so is something the sender wholly controls
without any new protocol machinery.

> sending a SACK for every packet, one sending it for every other with a
> delay ACK timer. We varied the timer.  The result was that the
> bandwidth was not shared equally. The association sending a SACK for
> every packet got the higher throughput. Using the I-Bit as described
> above mitigates this.  You can find more details in a paper with the
> title: Improving the Acknowledgment Handling of SCTP

(I can't find the paper ... can you send me a copy?)

The I-Bit mitigates what?  If you want the receiver to ACK every segment
then make it do so.  I don't quite follow this reasoning at all.
Delayed ACKs hurt performance, ACK-every helps it.  OK.  But, what ... ?
We want to pretend that the receiver does delayed ACKs, but have the
sender over-rule that notion and turn them on?  That doesn't make any
sense to me.

> SACK and the receiver has no idea.

OK- fair enough.  I'd call this a corner case that nobody would really
care about.  Generally speaking nobody sits around waiting on a
connection tear-down to move on to their next piece of work.

> > something should be *retransmitted*.  So, the samples you get will
> > be naturally taking into account any delay in the ACKing process and
> > the RTO should then accommodate that extra delay.
>
> I agree with you, it is a misconfiguration here. But I have seen this
> being deployed... The sender can detect this but can't do

The sender can detect this and do nothing?  What?  The sender is **in
control of** the RTO!  This is completely nonsensical.

> provides an API for getting notified when there is no outstanding

But, you're attacking that problem from the wrong side.  You should be
working on ways to move that data out of the sender faster, not worrying
about a small bit of added delay introduced by the receiver.  Or, *at
least* worry about the sending first and then if there are still issues
with the receiver you can then see about addressing those.  But, you're
not focused on the fundamental issue here.  ACKing immediately can save
a little time.  A smarter sending pattern can save RTTs.

> try to integrate some of the above into the ID to make it clearer.

Well, no.  It is no more clear, IMO.  And, yes, you should be able to
provide some crisp motivation in the document.  It seems to me that
without some actual motivation this should not be taken up.

allman

---

From: Michael Tüxen <[email-masked]>
Date: Thu, 25 Aug 2011 11:57:24 +0200

> without any new protocol machinery.
I agree. This was pointed out by Kacheong, that using an SCTP
specific variant of the Nagle algorithm resolves this issue.
Considering B/E bits in the Nagle algorithm is a similar change
compared to setting the I bit under that condition and is sender
side only.
> (I can't find the paper ... can you send me a copy?)
I'll send you one in a separate e-mail.
> sender over-rule that notion and turn them on?  That doesn't make any
The point is that it improves the situation to not delay *some* SACKs.
And only the sender knows which ones are the ones.

Simple example: After a timeout the cwnd = MTU. So you can send
a packet, the peer will delay the SACK, and the sender has to wait
for RTT + SACK.delay to send the next one. You can reduce this
to RTT.

I'm not saying that this makes a huge difference, but the mechanism
we are talking about is also not very complex...
> connection tear-down to move on to their next piece of work.
It is a matter of resources allocated at the server.
If you are running a server which handles lots of short living
(compared to 200 ms) associations, it can reduce the resources
substantially.
However, I agree, this applies only to specific services.
> control of** the RTO!  This is completely nonsensical.
Let me be more precisely: What I have seen: RTO.min = 50ms, RTO.max = 200ms
on the one side. These values are chosen to fulfill some failover
requirements. If the peer delays the ACK by 200ms, the pattern could observe
was: Send a packet, retransmit it, get a SACK (with the duplicate TSN listed).
The problem comes up whenever RTT + SACK.delay > RTO.min.

You pointed out that the sender measuring the RTT will include the SACK.delay
in its measurements. For SCTP this is only partially true.
RTT measurements are done using user data (where it does include the SACK.delay)
and via the heartbeat mechanism (where it does not include it).

However, when receiving the SACK with the duplicate report, it can
try to improve the situation. However, the solution of the problem
is to change the configuration of the receiver.

I don't consider this a major reason for introducing the I-Bit. It is
just a possibility to improve a bad situation if you have the
mechanism in place.
> a little time.  A smarter sending pattern can save RTTs.
I understand what you are saying.
But in the case of DTLS/SCTP we use multiple streams to decouple
traffic. However, when it comes to perform a DTLS handshake we
have to make sure that all messages fulfill some sequencing constraints,
no matter what streams they use.
This is done by stopping the sender, draining what it in flight, do
the handshake, and continue to send without any constraints anymore.
I can go into detail here, but you can also look it up in
http://tools.ietf.org/html/rfc6083

The pattern which is used is: Wait for sending the next packet until
the peer has acknowledged everything at the transport layer.
So you are waiting for the so called SENDER_DRY_EVENT.
If the SACK is delayed by 200ms, you wait 200ms longer than needed.
And again: The sender knows for which packets delaying the SACK
hurts.
It would be an alternative to switch delaying SACK off by default,
but this would result in more packets. So it might help with the
delay but is not optimal in another way.

Using delayed SACKs during the handshake really hurts performance
wise, since it can add up to a second.
> without some actual motivation this should not be taken up.
Is it now clearer, that there are situation where
* delaying the SACKs hurts for specific packets
* only the sender knows for which are these specific packets

Best regards
Michael

---

From: Mark Allman <[email-masked]>
Date: Thu, 25 Aug 2011 08:28:10 -0400

> I'll send you one in a separate e-mail.

(thanks!)

> And only the sender knows which ones are the ones.

I might have some sympathy for this approach if you were playing our
scenarios where this was clearly the case.  But, 

> I'm not saying that this makes a huge difference, but the mechanism we
> are talking about is also not very complex...

If you're talking about things like "optimizing" the **RTO** (which
triggers when the usual loss recovery doesn't work) you'll surely
understand why I am skeptical and unimpressed with the motivation,
right?! 

> It is a matter of resources allocated at the server.  If you are
> running a server which handles lots of short living (compared to 200
> ms) associations, it can reduce the resources substantially.  However,
> I agree, this applies only to specific services.

We might differ in our definition of "substantially" here.

> Let me be more precisely: What I have seen: RTO.min = 50ms, RTO.max =
> 200ms on the one side. These values are chosen to fulfill some
> failover requirements. If the peer delays the ACK by 200ms, the
> pattern could observe was: Send a packet, retransmit it, get a SACK
> (with the duplicate TSN listed).  The problem comes up whenever RTT +
> SACK.delay > RTO.min.
> 
> You pointed out that the sender measuring the RTT will include the
> SACK.delay in its measurements. For SCTP this is only partially true.
> RTT measurements are done using user data (where it does include the
> SACK.delay) and via the heartbeat mechanism (where it does not include
> it).
> 
> However, when receiving the SACK with the duplicate report, it can try
> to improve the situation. However, the solution of the problem is to
> change the configuration of the receiver.

Two things:

(1) So, to fix the sender's stupidity we're going to introduce not only
    new machinery, but machinery at both endpoints?  I don't think this
    approach is well thought out.

(2) Also, sure, not all RTT samples include the delayed ACK timer.  Not
    all data segments include it either (e.g., the second data segment
    triggers an immediate ACK and if that is the segment you're timing
    then you include no delay).  TCP has worked this way for a while.
    We have a variance for such things.

> I don't consider this a major reason for introducing the I-Bit. 

You keep saying such things.  What is the major reason?

> the peer has acknowledged everything at the transport layer.

This seems to be the recurring theme.  I.e., we are waiting for an ACK
to send something that could otherwise be sent.  Well, send it.  Why
involve the receiver in such situations?

> And again: The sender knows for which packets delaying the SACK

The *application* knows.  So, there is more complication here in that
the app has to communicate this to SCTP.

> It would be an alternative to switch delaying SACK off by default, but
> this would result in more packets. So it might help with the delay but
> is not optimal in another way.

It doesn't seem to me the delay is the problem in the cases you are
sketching.  It seems the sender's rule for when it is allowed to
transmit something are the issue.

The bottom line for me is that at the very least you need to actually
motivate the mechanism in the document.  The current list of motivations
is terse and glib.  And, I'd suggest you think all this through because
it seems to me from your explanations that you're nipping at a symptom
of a problem and not addressing the actual problem.  IMHO.

allman

---

From: Michael Tüxen <[email-masked]>
Date: Thu, 25 Aug 2011 15:31:55 +0200

> understand why I am skeptical and unimpressed with the motivation,
The above example is not related to any optimization of RTO, it
is about the case that the sender has to wait for for an incoming
SACK to send the next DATA  chunk because the cwnd = MTU.
>    approach is well thought out.
Why is the sender stupid if it has been configured with the above
values? The problem is that the configuration of the sender and
the receiver are not in tune.
> involve the receiver in such situations?
I can't, because it would break the application protocol.
But may be I'm overlooking something and you can suggest how
I can enforce some ordering over *all* streams without using
the SENDER_DRY_EVENT.

> the app has to communicate this to SCTP.
No problem. It uses the SCTP_SACK_IMMEDIATELY flag in the send call.
This is implemented in FreeBSD and Linux and described in section 6.
> transmit something are the issue.
For Nagle interaction: yes.
For the association teeardown: no.
For the application protocol interaction: I don't know an alternative.
That doesn't mean that there is any, but for me this is the main
motivation. This problem comes up when you move application protocols
running over TCP to SCTP, because they would benefit from multiple
streams. Often, you can decouple traffic streams for most of the time,
but at specific times, everything has to be synchronized over all
streams. Without changing the application protocol (by adding
an app level ack), the sender waits for all TSNs to be cum acked,
and the receiver has to process all messages. This is normally
sender driven. Examples are DTLS and SSH. The times where you need
to sync everything are the ones where you do some sort of rekeying.
> of a problem and not addressing the actual problem.  IMHO.
I agree that the motivations are not very detailed. This is why you
also have only MAYs on the sending side. 
The crucial point is that the receiver
* either ignores the I-Bit (because he isn't implementing it)
* or sends a SACK back.
As said above, this is implemented in Linux and FreeBSD.

The most important usage scenario is the app driven thing.
Depending on the service you run a server, the SHUTDOWN thing
might be important. But it really depends on the server and
is only related to resources.

So: Can you come up with a synchronization method which does not
use the SENDER_DRY_EVENT?

Then one could argue to not use the I-Bit. Except for the case
that implementations are out and it would be really bad if the
bit will be used from something different in the future.

Best regards
Michael

---

From: Mark Allman <[email-masked]>
Date: Thu, 25 Aug 2011 15:06:19 -0400

> The above example is not related to any optimization of RTO, it is
> about the case that the sender has to wait for for an incoming SACK to
> send the next DATA chunk because the cwnd = MTU.

I was careless.  I understand you are not optimizing the estimator.
But, you're trying to optimize the *process* of retransmitting via
timeout.  I don't find that compelling.

> values? The problem is that the configuration of the sender and the
> receiver are not in tune.

Thank you for making my point.  The sender has whacked things down far
enough that it is out of whack with the way a receiver works.  That
seems stupid to me.

> I can't, because it would break the application protocol.  But may be
> I'm overlooking something and you can suggest how I can enforce some
> ordering over *all* streams without using the SENDER_DRY_EVENT.

I had to go back and re-read.  I mis-understood what you were saying.  I
see your point here.  I guess.  I am not entirely sure I understand how
this is not a problem you created with the multiplexing.  But, I'll
certainly agree that I don't quite understand the DTLS process here.

> is only related to resources.

I don't think it is too much to ask you to develop the motivation
crisply in the document (and moreso *before* you think something should
be adopted by the WG).

> bit will be used from something different in the future.

Well, bluntly, that is stupid.  I could implement a lot of bad ideas and
use the "its implemented" excuse to ram it through the standards
process?  Give me a break.  (I mean, I know full well that works
sometimes, but its still stupid.)  The question came up as to whether
the WG should take this or not.  If the WG does not and some
implementation subsequently gets confused thats its own problem as far
as I am concerned.

allman

---

From: Michael Tüxen <[email-masked]>
Date: Fri, 26 Aug 2011 14:00:53 +0200

> timeout.  I don't find that compelling.
I want to say that there are situations in which only the senders
knows that not delaying the SACK helps to improve the protocol
behavior.
I do not think the we need an exhaustive list in the ID...
> enough that it is out of whack with the way a receiver works.  That
The point here is that the values used by the sender are the correct
ones (used in some cases for SIGTRAN setups). So it is not stupid
that the sender has been configured that way. The problem is the
misconfiguration of the receiver. But the sender has no way to change
that.
You might argue that the sender can change its RTO values to behave
well with the receiver. The problem is that the setup can not fulfill
the failover requirements for SIGTRAN. That is why the sender is configured
that way.

Again: The real solution is to configure the receiver correctly. I think
we agree on this. My point is only: The sender can not change the config
of the receiver (at not by any protocol mechanism / socket API
functionality I'm aware of), it can only mitigate the consequences
of the misconfiguration.
> this is not a problem you created with the multiplexing.  But, I'll
The only way to avoid this is to keep everything in sequence always.
But this means not using multiple streams.
The DTLS spec allows the DTLS layer to reorder packets to some degree,
but throws away packets being too late (either time based after
an change of the epoch or numbering based). Since SCTP provides a
reliable transport, we have to avoid that the DTLS layer throws
away the messages.

So yes, the problem is a consequence of using multiple streams.
The alternatives:
(a) Don't use multiple streams.
(b) Use a DTLS connection per bi-directional stream, similar to
    what http://tools.ietf.org/html/rfc3436 does.
We did not choose (a) because we wanted to support multiple streams.
The problem with (b) is the same as with the solution described in
RFC 3436: Scalability. Setting up a single SCTP association with
N bidirectional streams requires setting up N (D)TLS connections,
which is (in my sense) a substantial overhead (message wise,
CPU resources wise).

So if you agree that we should neither do (a) nor (b), you need
to suggest a better way to enforce some sequencing constraints,
which cover messages from all streams. We made use of the TSN,
which numbers all SCTP data chunks.
Again: I'm not able to come up with a better solution, but that
does not mean that there is nothing better.
> crisply in the document (and moreso *before* you think something should
Section 4 lists situations in which you might want to set the I-Bit.
I can take out the reason related to the Nagle algorithm since
it is much better to improve the Nagle algorithm. Not sure
if I would use the I-Bit if it is available on a platform, but
the Nagle code does not handle the stuff itself. This is
the case with current versions of FreeBSD, not sure about Linux.

I can add a section which describes somewhat more detailed
what is listed in section 4.1, if that helps.

However, I'm not sure if you

either want more text describing what is listed in section 4.1

or think adding a flag which makes it possible for the sender
to request a SACK to be sent back immediately is an idea
which is stupid/bad/worthless/whatever.
> use the "its implemented" excuse to ram it through the standards
Hmm. I don't think adding the I-bit is a bad idea. Actually I think
it should have been there right from the beginning. But I can
be wrong.
> implementation subsequently gets confused thats its own problem as far
I'm concerned about interoperability of the major Unix based implementations.
However, it is up to the working group to decided if it is interested
in the work or not. Whether or not the working group takes into account
if the functionality is available in current versions of OSes is also
up to the WG. But I wanted to make sure that it is known that it has
been implemented and released. That's why I can make a statement
about complexity (based on the FreeBSD implementation).

Best regards
Michael

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Fri, 26 Aug 2011 21:27:40 -0600

Michael,

Michael Tüxen wrote:                                               (Fri, 26 Aug 2011 14:00:53)
> functionality I'm aware of), it can only mitigate the consequences

I agree with Mark, RTO.max < 200ms without setting SACK.delay at the
peer (even to zero) is plain stupid, even for SIGTRAN.

Also, there is no need for a per-message SACK immediately mechanism in
such a case.

At most a mechanism to negotiate maximum SACK.delay on association
establishment.  Also, it would be better to negotiate the SACK.delay
to lower than 200ms, but not to zero.

Also, for most SIGTRAN applications, SACK.delay is not often a
consideration as many messages are sent back-to-back in a continuous
stream and a SACK will be sent every second DATA chunk without
SACK.delay expiring at all.  It is highly unlikely that even a 50ms
quiet period will occur in practice, and the odd suprious retransmission
that would result is not a issue.

--brian

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

From: Michael Tüxen <[email-masked]>
Date: Sat, 27 Aug 2011 12:25:04 +0200

> peer (even to zero) is plain stupid, even for SIGTRAN.
As said earlier: It is a mis-configuration. The solution to it
is the reconfigure the receiver. My only point was that there
is currently no way for the sender to change the SACK.delay of the
receiver.
> Also, there is no need for a per-message SACK immediately mechanism in
I don't think that mitigating the consequences is the reason why
you want to introduce the I-bit. I just wanted to say, that,
if you have it, you can use it to mitigate the consequences.
However, there would be no need to do it per-message base,
you would set it on all DATA chunks.
> to lower than 200ms, but not to zero.
That would be a solution.
> that would result is not a issue.
It depends on the upper layer and the service provided.

However, I think considering this mis-configuration case adds
more confusion than the use of the I-bit on that case might help.
I'll remove it in the next rev. of the ID.

Best regards
Michael