Subject: Re: [Tsvwg] A question of Un-Ordered SCTP

From: Randall Stewart <[email-masked]>
Date: Wed, 12 Feb 2003 06:25:26 -0600

This looks good.. but would it be
better to make the FEATURE_ID 16 bits..

.. my reasoning is that cookies are getting
bigger and it might be best to control there
length to some degree... I could go either way
but I think 65,535 features is more than enough
to choose from :>

R

Tuexen Michael wrote:
> Randy,
> 
> I think having a FEATURES parameter would be a good idea. I do not
> like the idea of bitmask. Instead of this I would propose something
> like a FEATURES parameter which looks like:
> 
>        0                   1                   2                   3
>        0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
>       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>       |   FEATURES_PARAMETER_ID       |       Parameter Length        |
>       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>       |                         FEATURE_ID 1                          |
>       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>       |                         FEATURE_ID 2                          |
>       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>       |                         FEATURE_ID 3                          |
>       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>       |                         FEATURE_ID 4                          |
>       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> 
> Each feature will be an IANA assiged 32 bit uint. This is extendable.
> 
> Best regards
> Michael
> 
> Michael Tuexen   Tel.:   +49 89 722 47210
> Siemens AG       Fax:    +49 89 722 48212
> ICN CP SE C 7    e-mail: [email-masked]
> 
> 
>>-----Original Message-----
>>From: Randall Stewart [mailto:[email-masked]]
>>Sent: Wednesday, February 12, 2003 12:59 PM
>>To: [email-masked]; Michael Tuexen
>>Cc: Caitlin Bestler; [email-masked]; Tuexen Michael; [email-masked]
>>Subject: Re: [Tsvwg] A question of Un-Ordered SCTP
>>
>>
>>Brian/Michael:
>>
>>I do like the idea of keeping things the same (aka same
>>methodology)... this is good.
>>
>>It is also neat to be able to tell the difference between
>>an implementation that supports this feature and one that
>>does not.  But there are positives and negatives, I think,
>>to each idea...
>>
>>A) For using a TLV, (as Michael suggests)
>>    we really need to put this into a seperate
>>    draft.. when I do think one could easily fit
>>    this into the IG as Brian as suggested.
>>
>>B) montonically increasing numbers would eventually
>>    wrap when the SSN hit FFFF back to 0... thus
>>    the application would need to have some sort
>>    of state to track if the peer supported the
>>    feature.
>>
>>It seems to me it is only the first message that
>>arrives that benefits from the start at 0. If the
>>application assumes a "not supported", then by
>>starting the SSN at 1 the app can flip the flag
>>from "not supported" to "supported" on the arrival
>>of the first message.
>>
>>If the application instead set its flag to "supported"
>>the first message would not tell this information and
>>instead it would only be upon the second arriving message
>>that the app could make the flag determination... causing
>>either a delay in delivery of the first message (yuck) or
>>just toss it up and wait.
>>
>>More importantly, I am wondering, if the implementation does
>>NOT support this, what will the app do? Change its
>>message structure? If so it will need to have some clues
>>(as the initiator) that its implementation does not
>>support the feature .. possibly a HAVE_SCTP_UNORDERED_SSN
>>in sctp.h... but the problem is you would not know about
>>the other guy until its first unordered message arrives.
>>So if you have to send the first message what do you do?
>>
>>if HAVE_SCTP_UNORDERED_SSN is 0.. no problem.. you
>>send a message that has the imbedded sequence number.. but
>>what do you send when HAVE_SCTP_UNRODERED_SSN is set to 1?
>>
>>If you send the imbedded sequence in the message anyway whats
>>the point of this little trick? The second message (which you
>>may have to send before knowing receiving a unordered
>>msg from the peer) would still have this, and maybe the third
>>etc...
>>
>>Hmm.. the more I think on this the more it becomes clear that
>>you must know ahead of time i.e. when the association comes
>>up to "do the right thing".... which goes back to Michaels
>>idea of a TLV.
>>
>>I have been wondering for quite some time now if we don't
>>just need a TLV in the INIT/INIT-ACK that indicate features
>>in general... something like a bit mask or byte array indicating
>>each feature...
>>
>>addip
>>delip
>>prsctp
>>setprimary
>>unordered-assigns-sequences
>>
>>etc..
>>
>>Maybe this is a better way to go.. and we could write a short
>>draft dictating this bit/byte TLV... code as a 0x80 class.. that
>>way old implementations will throw it away... (we really don't
>>care if they don't understand it)...
>>
>>I have been avoiding this.. but it might be the best approach..
>>
>>Then again, I guess it could also be put in addip or prsctp since
>>it would effect them as well :> Then one could put the
>>other change in the IG and thus have a very easy change that
>>is in the IG and a minor update to either addip or prsctp...
>>
>>
>>What do you all think?
>>
>>R
>>
>>Brian F. G. Bidulock wrote:
>>
>>>Randall,
>>>
>>>I agree to having SCTP set the sequence number and, as you said in
>>>your first note, SCTP passing the sequence number to the ULP on
>>>the receiving side is already done for ordered traffic.  Some even
>>>pass the zero now.
>>>
>>>I suppose that SCTP would set this sequence number when it is given
>>>the data for delivery at the sending ULP and the sequence 
>>
>>would represent
>>
>>>the order that the data was presented to SCTP by the ULP at 
>>
>>the sending
>>
>>>side.
>>>
>>>I think it would be good to see the requirement that the sequence
>>>number begin at 1 and be monotonically increasing in the 
>>
>>saem fashion
>>
>>>as ordered streams.
>>>
>>>That way RDDP and other applications could perform their 
>>
>>own reordering
>>
>>>at the receiver.  I think it would be a good thing to have SCTP set
>>>this value.
>>>
>>>In fact, I can see another application for this feature.  M2PA
>>>(SIGTRAN protocol) needs in some circumstances to send data 
>>
>>unordered
>>
>>>with the hope of expedited delivery (does not want control 
>>
>>information
>>
>>>stuck behind large amounts of data in an ordered stream), 
>>
>>however, the
>>
>>>order of processing of the expedited control information is 
>>
>>important.
>>
>>>This facility would permit M2PA to send such information unordered
>>>to expedite it and the receiver could perform the reordering itself.
>>>Up until now, the only solution was to place a sequence 
>>
>>number in with
>>
>>>the control information.
>>>
>>>Could we perhaps add a section to the SCTP IG.  "Problem statement:
>>>the sender wishes to send unordered data but the receive 
>>
>>needs to know
>>
>>>the original order of sending..." or some such.
>>>
>>>It should be quite simple to add the feature to existing SCTP
>>>implementations.
>>>
>>>--brian
>>>
>>>
>>>On Tue, 11 Feb 2003, Randall Stewart wrote:
>>>
>>>
>>>
>>>>Michael:
>>>>
>>>>This seems like a valid method.. it gets around Brian's objection
>>>>(which was a good point) and yet still provides the application with
>>>>a hint about what the assigned sequence order was on the unordered
>>>>messages...
>>>>
>>>>R
>>>>
>>>>Michael Tuexen wrote:
>>>>
>>>>
>>>>>Caitlin,
>>>>>
>>>>>thank you very much for the clarification.
>>>>>
>>>>>I do not like the idea to pass the SSN transparently from 
>>>>
>>the ULP to the 
>>
>>>>>ULP.
>>>>>But having two sequence spaces is OK for me. This means 
>>>>
>>that the SSN at the
>>
>>>>>sending side is controlled by the SCTP and given to the ULP at the 
>>>>>receiving side.
>>>>>
>>>>>This would be a clean solution. So I voting for have two 
>>>>
>>separate number 
>>
>>>>>spaces
>>>>>for user data. Since the one for ordered delivery exists 
>>>>
>>already in RFC 
>>
>>>>>2960, this
>>>>>only means that a new one is introduced for the unordered 
>>>>
>>delivery user 
>>
>>>>>data.
>>>>>
>>>>>Best regards
>>>>>Michael
>>>>>
>>>>>On Tuesday, Feb 11, 2003, at 17:37 Europe/Berlin, Caitlin 
>>>>
>>Bestler wrote:
>>
>>>>>
>>>>>>On 2/11/03, Tuexen Michael wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>>>Dear all,
>>>>>>>
>>>>>>>does this mean that you want two sequence spaces per
>>>>>>>stream: one for the 'ordered' user data and one for the
>>>>>>>unordered user data?
>>>>>>>
>>>>>>
>>>>>>The RDDP/SCTP adaptation only uses unordered delivery, so
>>>>>>that question did not really arise.
>>>>>>
>>>>>>But if you were to generalize the problem, I believe the
>>>>>>answer would be yes.
>>>>>>
>>>>>>The SSN for ordered delivery would be unchanged. The
>>>>>>addition would be providing "advisory ordering" for
>>>>>>unordered delivery. The general characteristic is that
>>>>>>the receiving ULP wants the data in whatever order it
>>>>>>arrives, but wants to know what the original order was.
>>>>>>
>>>>>>In addition to RDDP, RTP over UDP is another example of
>>>>>>an application that wants to be aware of the original
>>>>>>ordering, even while accepting the data out-of-order.
>>>>>>
>>>>>>Caitlin Bestler
>>>>>>http://asomi.com/CaitlinBestler/
>>>>>>_______________________________________________
>>>>>>tsvwg mailing list
>>>>>>[email-masked]
>>>>>>https://www1.ietf.org/mailman/listinfo/tsvwg
>>>>>>
>>>>>
>>>>>_______________________________________________
>>>>>tsvwg mailing list
>>>>>[email-masked]
>>>>>https://www1.ietf.org/mailman/listinfo/tsvwg
>>>>>
>>>>>
>>>>
>>>>
>>>>-- 
>>>>Randall R. Stewart
>>>>[email-masked] 815-342-5222 (cell phone)
>>>>
>>>>_______________________________________________
>>>>tsvwg mailing list
>>>>[email-masked]
>>>>https://www1.ietf.org/mailman/listinfo/tsvwg
>>>
>>>
>>
>>-- 
>>Randall R. Stewart
>>[email-masked] 815-342-5222 (cell phone)
>>
> 
> 
> 

-- 
Randall R. Stewart
[email-masked] 815-342-5222 (cell phone)

_______________________________________________
tsvwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tsvwg

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Wed, 12 Feb 2003 05:31:57 -0700

Randall,

For RDDP why not just suppress ordered delivery (order of arrival rather than
order of sending) of data in the stream at the receiver.  That way you can use
the SSN of ordered DATA chunks instead of trying to add SSNs to unorderd data
chunks.

Also, that way a RDDP/SCTP compliant sender will be compatible with a
non-RDDP/SCTP compliant receiver.

If you use unordered DATA chunks, the non-RDDP/SCTP compliant receiver will
deliver data out of order (instead of in order) on the stream, which is a
bad thing.

--brian

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

From: Caitlin Bestler <[email-masked]>
Date: Wed, 12 Feb 2003 10:25:19 -0600

On 2/12/03, Brian F. G. Bidulock wrote:

>Randall,
>
>For RDDP why not just suppress ordered delivery (order of
>arrival rather than order of sending) of data in the
>stream at the receiver.  That way you can use the SSN of
>ordered DATA chunks instead of trying to add SSNs to
>unorderd data chunks.
>
One of RDDP's requirements is that data be provided to it as
soon as possible. (Regular readers of this mailing list will
note how carefully I avoided using the word "delivered").

So, trying to work with SCTP on SCTP's terms, you have three
possible strategies:

-- Include a SSN-equivalent in the Data Chunk.

-- Allow out-of-order placement of ordered Data chunks.

-- Make SCTP's SSN meaningful for unordered delivery.

The pure ULP solution obviously has the least impact on
the SCTP layer. But it smacks of having the ULP duplicate
functionality that rightfully belongs at the transport
layer. I believe the quick consenseus is that sequenced
unordered delivery is a natural concept that has value
beyond any single ULP.

Allowing out-of-order placement of ordered Data chunks
is possible, but strikes me as a major change to an SCTP
reciever. To achieve the desired performance, the  new
feature must be integrated with the existing SCTP stack,
and would likely require it to be restructured. NOT
desirable.

The last option is clearly easily implemented.

This is particularly true if you adopt Randall's suggested
algorithm for identifying traditional "no sequence
information" unordered delivery by the fact that you have
*never* gotten a non-zero SSN on an unordered packet.

When a receiveer has *never* gotten a non-zero SSN on an
unordered packet, it may only infer order from the TSN.
That algorithm will work for RDDP.

It even allows RDDP to be built on top of an unmodified SCTP
stack, which is a very good thing. The impact of not
supplying the SSN is that the RDDP receiver can only
determine that a Data chunk is deliverable when *all*
Data Chunks with lower TSNs have been delivered, whether
they are part of the same stream or not. Given that one
of the goals of SCTP is to avoid head-of-line blocking
that didn't strike me as a complete solution.

>Also, that way a RDDP/SCTP compliant sender will be
>compatible with a non-RDDP/SCTP compliant receiver.
>
>If you use unordered DATA chunks, the non-RDDP/SCTP
>compliant receiver will deliver data out of order (instead
>of in order) on the stream, which is a bad thing.
>

The ULP is either RDDP or it is not. If the sender and
receivers are not in agreement about what ULP is in use
then you already have a bit of a problem.

Caitlin Bestler

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Wed, 12 Feb 2003 10:14:51 -0700

Caitlin,

> and would likely require it to be restructured. NOT

Not true.  Ordered delivery takes work.  Order of arrival
delivery does not.  SCTP does both.  Altering the SCTP
receiver merely consists of treating non U-bit Data Chunks
like U-bit Data Chunks with SSNs.  The SCTP receiver already
has all the necessary mechanisms.

> The last option is clearly easily implemented.

Not easier than allowing out of order placement of ordered
data chunks.

Implementations have been allowed to make the assumption that
an SSN does not exist for U-bit data chunks.  They already
exist for ordered data chunks.

The second approach requires that the SCTP transmitter start
numbering unordered data chunks with a separate numbering
space.  This required another expensive 16-bits per stream
structure.  When supporting thousands of streams this is
expensive, particularly if unused on a given association.

Delivering unordered data chunks as they arrive is far
less expensive.  If done on a per-association basis,
no per-stream structure increases are necessary.  Even if
doen on a per-stream basis, only one (probably already
unused) flag bit per stream would be necessary.

> That algorithm will work for RDDP.

The algorithm does not help per-stream structure expense.
The extra 16-bits per stream must be allocated in preparation
for such a delivery.

> stack, which is a very good thing.

The other approaches permit that too.

> The impact of not
> they are part of the same stream or not.

The second approach has the efficiency that (if unmodified)
the data will be delivered ordered and RDDP can place them
faster than if SSN is not included in the unordered data
chunks.  This is because data chunks will be delivered
around missing TSNs from other streams.

By working with TSNs you would re-introduce inter-stream
HOL blocking which the second approach would eliminate.

> Given that one

A receiving ULP could be configured to ignore the RDDP
header in received data.

The first approach does not require any SCTP modifications.

The second approach (allowed unordered delivery of ordered
data chunks) requires the least expense on the part of the
SCTP implementation (no per-stream structure increases) and
uses existing procedures (unordered delivery).

The third approach requires the most expense (per-stream
structure increases) and delivery of SSN to the ULP where
it was not previously delivered.

--brian

-- 
Brian F. G. Bidulock    ¦ The reasonable man adapts himself to the ¦
[email-masked]    ¦ world; the unreasonable one persists in  ¦
http://www.openss7.org/ ¦ trying  to adapt the  world  to himself. ¦
                        ¦ Therefore  all  progress  depends on the ¦
                        ¦ unreasonable man. -- George Bernard Shaw ¦
_______________________________________________
tsvwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tsvwg

---

From: Caitlin Bestler <[email-masked]>
Date: Wed, 12 Feb 2003 14:28:40 -0600

On 2/12/03, Brian F. G. Bidulock wrote:

>> 
>> It even allows RDDP to be built on top of an unmodified
>> SCTP stack, which is a very good thing.
>
There is a distinction when running over unmodified SCTP
stacks.

With sequenced unordered delivery, an DDP receiver will
have the Data Chunks delivered to it even with no
modifications to either SCTP stack. In fact it will probably
get the non-zero SSN if the sender's SCTP stack is the only
one that has been modified.

But even if there is no additional information, the data may
be placed immediately. It is only final delivery that is
subject to cross-stream head-of-line blocking.

With a modified stack, early placement of ordered Data
Chunks clearly fulfills the needs of RDDP. However, if the
SCTP receiver is unmodified, in-stream drops will result
in HOL blocking of placement. This will require intermediate
buffering of the Data Chunks, which RDDP seeks to avoid.

A scenario where unordered delivery is a problem:

    TSN: 101    Stream ID: 7    SSN: 0
    TSN: 102    Stream ID: 8    SSN: 0
    
    TSN: 104    Stream ID: 7    SSN: 0
    
When the Data Chunk with TSN 104 is received, it can
be placed, but it cannot be delivered to the ULP because
there is no way to know which Stream ID TSN 103 was for.

Adding the SSN, solves that. In the following example it
would be clear that TSN 104/SSN 18 is in fact deliverable.

    TSN: 101    Stream ID: 7    SSN: 17
    TSN: 102    Stream ID: 8    SSN: 43
    
    TSN: 104    Stream ID: 7    SSN: 18

The problem with ordered delivery over an unmodified SCTP
receiver deals with the following scenario:

    TSN: 101    Stream ID: 7    SSN: 17
    
    TSN: 103    Stream ID: 7    SSN: 19
    
An unmodified SCTP receiver will not make TSN-103/SSN-19
available to the DDP layer until TSN-102/SSN-18 shows up.
The impact of the in-stream gap is that non-ULP buffering
is required for post-gap Data Chunks. This is a far greater
impact that the delay in delivery of the first scenario.
One deals with buffering capacity and memory bandwidth, 
the other deals with latency.

Clearly, either modification solves the problem. But I
believe unordered delivery is clearly a better solution
when working with unmodified SCTP stacks.

Caitlin Bestler

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Wed, 12 Feb 2003 13:43:04 -0700

Caitlin,

> one that has been modified.

Or probably not, because there was no need to deliver SSN=0
for each unordered data and it would be inefficient to do so.

> subject to cross-stream head-of-line blocking.

So?  The time taken to place the last in-order pieces before
indicating final delivery is probably to the time taken to
place the last out-of-order piece before final delivery.

> there is no way to know which Stream ID TSN 103 was for.

Using ordered data chunks solves that.

> available to the DDP layer until TSN-102/SSN-18 shows up.

So what?  DDP cannot indicate delivery until SSN=18 shows
up either.  All it can do is place SSN=19 before SSN=18
which is hardly a boost.  It still has to place SSN=18 in
the gap and the unmodified ordered data chunks scenario is
far better than the unmodified unordered data chunks scenario.

> impact that the delay in delivery of the first scenario.

It is not.  Replace last your example above wtih SSN: 0 on
TSN 101 and TSN 103.  The unmodified unordered data chunk
scenario performs as poorly in that specific case than the
unmodified ordered data chunk scenario.

The unmodified ordered data chunk scenario performs better
than the unmodified ordered data chunk scenaio in the first
example: adding SSNs:

     TSN: 101    Stream ID: 7    SSN: 20
     TSN: 102    Stream ID: 8    SSN: 12
     
     TSN: 104    Stream ID: 7    SSN: 21

The unmodified ordered data chunk secnario will deliver TSN
104 immediately.  The unmodified unordered data chunk scenario
will have to wait for TSN 103 before it discovers that it
could have placed and delivered TSN 104.

The unmodified ordered data chunk scenrio is always equal to
or better than the unmodified unordered data chunk scenario
because the first can detect SSN gaps.

> when working with unmodified SCTP stacks.

Quite the reverse is the case for unmodified stacks.

--brian

---

From: Tuexen Michael <[email-masked]>
Date: Wed, 12 Feb 2003 13:11:19 +0100

Randy,

       0                   1                   2                   3

Best regards
Michael