Subject: [Tsvwg] SCTP: using streams

From: "Denio Mariz" <[email-masked]>
Date: Wed, 7 Nov 2001 13:07:21 -0200

Hello,

We know the ability of SCTP to deal with more than one stream in the same
association.
But, I'm in doubt about the use of this ability in a real situation. So I
have two questions.

Supposing an application A is sending and receiving data to/from another
application B through an association (in a client/server situation):
a) In which case this communication would require more than one stream to be
used ? could you give a scenario ?
b) The initiative for create/use a new stream is taken by the client at the
time of sending a new message (through the SEND ULP-to-SCTP interface,
please see 10.1.E in RFC 2960). Am I correct ?

Thanks,

Denio Mariz

_______________________________________________
tsvwg mailing list
[email-masked]
http://www1.ietf.org/mailman/listinfo/tsvwg

---

From: Phillip Conrad <[email-masked]>
Date: Wed, 07 Nov 2001 17:41:57 -0500

>used ? could you give a scenario ?

In addition to Ivan's comments (sent in a another reply) also see the 
papers at the following links for two examples of applications that can 
take advantage of multiple streams, and a discussion of the performance 
benefits of doing so:

P. Conrad G. Heinz A. Caro P. Amer J. Fiore, "SCTP in Battlefield 
Networks", IEEE MILCOM 2001, Washington, DC, 10/01
http://www.cis.udel.edu/~amer/PEL/poc/pdf/milcom01-conrad-sctp.pdf

P. Conrad, A. Caro, P. Amer, "ReMDoR: Remote Multimedia Document Retrieval 
over Parital Order Transport", ACM Multimedia 2001, Ottawa, Canada, 
9/01   http://netlab.cis.temple.edu/techrpts/TUCIS-TR-2001-003.pdf

>please see 10.1.E in RFC 2960). Am I correct ?

Actually, the number of streams is fixed and must be specified at 
Association Establishment time, so there is no operation to "create" a new 
stream.   However, your main point is essentially correct: each time  the 
ULP (that is, the Upper Layer Protocol, the "user" of SCTP) requests to 
SEND data, one of the parameters to the SEND request is the stream number 
on which the outgoing message should be sent.

For some particular implementation, there may be a "default" stream number, 
but if so, that's an API issue, not a protocol issue.

Phill Conrad

Phill Conrad -- Asst. Professor, CIS Dept, Temple University

NOTE: For faster replies, please reply to [email-masked], NOT 
to [email-masked].

---

From: "DJamel H. Sadok" <[email-masked]>
Date: Thu, 8 Nov 2001 07:51:47 -0200 (EDT)

Hi,

Assuming that the point of having different concurrent streams is also
the possibility of routing them along different paths, I cannot see how
SCTP would make this task explict to IP.

Thanks, DJamel

---

From: Randall Stewart <[email-masked]>
Date: Thu, 08 Nov 2001 05:24:18 -0600

> SCTP would make this task explict to IP.

Routing streams on different paths has NEVER been the goal of
SCTP. The Streams were put in place to combat Head-Of-Line blocking.

R

-- 
Randall R. Stewart
[email-masked] 815-342-5222 (cell phone)

---

From: "Phillip T. Conrad" <[email-masked]>
Date: Thu, 08 Nov 2001 12:17:22 -0500

At 05:24 AM 11/8/2001 -0600, Randall Stewart wrote:
>SCTP.

In fact, if I understand the history correctly (Randy may correct me if I 
misspeak), routing different SCTP streams over different paths could be 
potentially harmful since the effects on congestion in the "big I" Internet 
are unknown at this stage.  While SCTP is multi-homed, only one interface 
is considered "primary" at any given time (though secondary interfaces may 
be used for retransmissions.)

Considerable additional research and experience would be needed before this 
would be considered feasible.

>  The Streams were put in place to combat Head-Of-Line blocking.

Head-of-line blocking, in this case, at the receiving transport layer 
entity (as opposed to the HOL blocking that occurs in certain types 
of   link and network layer switches.)

Specifically, the notion here is that when an application has multiple 
"logical" streams that are being multiplexed onto a single ordered 
transport connection, a packet (y) from stream (i) that has arrived at the 
transport receiver and is the "next expected packet" in stream (i), should 
not be held back from being delivered by a missing segment (x) on stream 
(j), i!=j, just because the missing segment (x) might have a lower sequence 
number than (y) in the single ordered stream.   In a fully ordered 
transport protocol such as TCP, since x has a lower "overall" sequence 
number than y,  it must be delivered before (y).

For example, if two jpeg images are being sent over a single TCP connection 
(either back-to-back, or interleaved; it doesn't really matter), a missing 
segment containing bytes from image one can end up causing delay in the 
delivery of segments containing bytes from image two.   This is an 
unnecessary delay, since the two streams are logically independent from the 
point of view of the application.   This is one of the arguments made for 
what has come to be known as Application Layer Framing (ALF).  The seminal 
paper on ALF is by Clark and Tennenhouse and appeared in SIGCOMM 1990.   In 
addition to the work I did at UD and Temple in this area, there is also 
work by Diot and Gagnon, among others. For full citations and more 
information, see the papers at the links I sent in a previous message.

Phill Conrad

---

From: Joe Touch <[email-masked]>
Date: Thu, 08 Nov 2001 17:07:19 -0800

Phillip T. Conrad wrote:

> entity (as opposed to the HOL blocking that occurs in certain types of   
> link and network layer switches.)

Seems that at least one solution is to use different transport layer 
associations, rather than to mux at the transport layer.

> transport connection, a packet (y) from stream (i) that has arrived at 
> the transport receiver and is the "next expected packet" in stream (i), 
> should not be held back from being delivered by a missing segment (x) on 
> stream (j), i!=j, just because the missing segment (x) might have a 
> lower sequence number than (y) in the single ordered stream.   In a 
> fully ordered transport protocol such as TCP, since x has a lower 
> "overall" sequence number than y,  it must be delivered before (y).

I.e, HOL blocking occurs if you mux over TCP.

So why mux at all ;-) ??

Joe

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Fri, 9 Nov 2001 00:47:30 -0600

Joe,

SCTP does not fully address HOL blocking, anyway, for several
reasons:

  - if records are larger than the association MTU, loss of packet
    on any stream blocks delivery on other streams that are
    mid-record.  This is because SCTP cannot tell whether the
    missing TSN was a segment in the middle of a record for a
    stream.  DATA chunks only have first/last bits and do not
    indicate the segment number as do other protocols. 

    (So you better break your jpeg files down into smaller than
    association MTU records.)

  - in contrast to multiple transport layer associations, muxed
    streams share a single congestion window.  This means that when
    a packet is lost, the congestion window for all streams closes.
    The closing of the congestion window will block packets from
    other streams at the transmitter.  When multiple transport
    associations are used, each has their own congestion window and
    when a packet is lost on one transport association, only that
    transport association's congestion window is closed.

  - when a packet is lost from one stream, the receiver will GAP
    ACK, however, because GAP ACKs can be reneged, the transmitter
    cannot clear packets from retransmission for any packet after
    the lost packet (regardless of stream).  This causes blocking at
    the transmitter, because the transmit buffer fills until the
    lost packet is cumulatively acked.

Considering these queueing effects, it is always better to have
multiple SCTP associations with a single stream than it is to have
one SCTP association with multiple streams.

You are correct.  SCTP does not solve HOL blocking in the wider
sense of the concept of queuing delays experienced by one stream of
data resulting from packet loss in another.

There is no incentive here to mux at all.

--brian

-- 
Brian F. G. Bidulock    ¦ The reasonable man adapts himself to the ¦
[email-masked]    ¦ world; the unreasonable one persists in  ¦
http://www.openss7.org/ ¦ trying  to adapt the  world  to himself. ¦
                        ¦ Therefore  all  progress  depends on the ¦
                        ¦ unreasonable man. -- George Bernard Shaw ¦

---

From: Joe Touch <[email-masked]>
Date: Thu, 08 Nov 2001 22:58:36 -0800

>     association MTU records.)

Sounds like a reason not to mux, again. Separate streams don't have that 
problem.

>     streams share a single congestion window.

See RFC2140 or the CM proposal. They _can_ share a window, depending on 
the implementation; i.e., muxing isn't the only way to achieve sharing.
If you _want_ to share fate, then you can; if you don't, you needn't.

>     lost packet is cumulatively acked.

Again, a reason not to mux.

> There is no incentive here to mux at all.

Agreed :-)

Joe

---

From: Randall Stewart <[email-masked]>
Date: Fri, 09 Nov 2001 07:02:37 -0600

> Sounds like a reason not to mux, again. Separate streams don't have that

Well, I fail to see the issue. Here is why:

1) When I do a send() to a stream the amount of data I send
   is considered a record (see the sctpsocket draft).
2) I doubt a send(huge-3meg-file) to a stream will suceed. The
   O/S ususally just does not have this kind of buffer space.
3) A far more likely thing that would be done is, in a thread,
   while(fread(data,block-size)){
     send(data,block-size);
   }
   Where your block size will conform to a FS boundary of some
   sort.. 1k, 2k or 4k...

  I just don't see it as an issue..

> If you _want_ to share fate, then you can; if you don't, you needn't.

I think from the standpoint of congestion control we WANT to share
fate. This is one of the reasons the CM as been developed. Being
able to get an unfair share of the bandwidth is NOT a plus. So
I don't see this as a reason to NOT use SCTP. In fact just the
opposite. Either SCTP or CM should be used, maybe both :>

> Again, a reason not to mux.

More of a reason not to reneg.. I know Chip and I were against
this as well, neither one of us liked reneging but it was the
only way to avoid a deadlock :-0

Maybe we should have put some nice bits in the SACK that told
if the TSN had been delivered... 

I had raised this point when we debated it, eons ago it seems,  but
it was thought that the small amount of sender side resources was
not that big of deal.. since you are trading off sender resources
versus receiver side resources... a trade off that does not
necessarily go away with multiple connections either... the
constraints just get pushed around a bit... 

> > There is no incentive here to mux at all.

I dis-agree. :-)

R

---

From: Phillip Conrad <[email-masked]>
Date: Fri, 09 Nov 2001 08:46:14 -0500

> > >     association MTU records.)

Making ADUs on the order of path MTU size records is part of the ALF 
principle anyway.

> > Sounds like a reason not to mux, again. Separate streams don't have that

Both Joe and Brian failed to note that the overhead involved in keeping 
multiple associations, both in terms of packets on the wire for connection 
establishment, as well as in terms of sender and receiver state in the TCB, 
significantly exceeds the overhead involved in having multiple 
streams.  This is a reason TO mux.

>opposite. Either SCTP or CM should be used, maybe both :>

Where can I find an open source (or for that matter, a proprietary) 
implementation of RFC2140 or the CM proposal?

>if the TSN had been delivered...

Yes, we perhaps should have... and perhaps still can as an extension.

> > > There is no incentive here to mux at all.

No one has so far addressed the overhead of connection establishment 
packets for multiple associations.

Reasonable people may disagree about the tradeoffs involved.  However, 
clearly, the statement that there is NO incentive to mux is NOT 
true.  There are aspects of SCTP that can be improved, and constructive 
suggestions would be more helpful than overstating the case.

>I dis-agree. :-)

I tend to disagree also.   However, what would be very helpful would be 
some benchmarking of  of RFC2140 with multiple TCP connections vs. a single 
SCTP association with multiple streams.  If someone can point me to an 
implementation of RFC2140, I'd be happy to conduct such benchmarks and 
publish the results.    If we can establish through *data* that the 
putative benefits of RFC2140 and multiple TCP connections outweigh those of 
SCTP's multiple streams, I'd be pleased to report such results.

If SCTP's multiple stream feature does not do what it was designed to do, 
such studies could point the way towards fixes.

Phill Conrad

---

From: Joe Touch <[email-masked]>
Date: Fri, 09 Nov 2001 08:40:12 -0800

Phillip Conrad wrote:

> multiple associations, both in terms of packets on the wire for 
> connection establishment, as well as in terms of sender and receiver 
> state in the TCB, significantly exceeds the overhead involved in having 
> multiple streams.  This is a reason TO mux.

1) syn packets aren't that big (I missed it - is SCTP only for low-speed 
links?)

2) there is no difference in the overhead to coordinate TCBs or to 
coordinate muxing within an SCTP stream. TCBs aren't that large, and the 
benefits (in terms of overall simplification, or avoiding complexity, of 
the protocol) are.

> implementation of RFC2140 or the CM proposal?

That's no reason to create a complex protocol, first.

Second, see http://nms.lcs.mit.edu/software/CM/

The RFC2140 work was implemented in ns for analysis, the code of which 
is available at http://www.isi.edu/~larse/software/index.html

> packets for multiple associations.

Largely because it's too small to matter in most cases. In cases where 
it really does matter, it is trivial to compress K connection requests 
into a single request (e.g., on a new port), and have it 'back-end' the 
uncompressed set back. This aggregation of requests and responses can be 
implemented trivially, ONLY for the connection establishment, if you're 
really that BW poor.

> some benchmarking of  of RFC2140 with multiple TCP connections vs. a 
> single SCTP association with multiple streams.  If someone can point me 
> to an implementation of RFC2140, I'd be happy to conduct such benchmarks 
> and publish the results.    If we can establish through *data* that the 
> putative benefits of RFC2140 and multiple TCP connections outweigh those 
> of SCTP's multiple streams, I'd be pleased to report such results.

The posted ns code might go a long way to start this; presumably there's 
an ns of SCTP that can be used for comparison.

Joe

---

From: Janardhan Iyengar <[email-masked]>
Date: Fri, 9 Nov 2001 12:26:19 -0500 (EST)

Hi all,

>    Where your block size will conform to a FS boundary of some

I am inclined to think that an ADU size > 1500 bytes is likely to get
fragmented (considering the more common packet sizes on the Internet)..

A (maybe naive) question:
Why are SSNs kept unique per ULP message, and not per chunk? If the reason
is so that the receiving SCTP can reassemble messages, then the B and E
flags (alongwith the per chunk SSNs) are sufficient for the same. And if
SSNs are kept unique per chunk, then the above mentioned problem of HOL
blocking can be eliminated.

regards,
jana

-----------------------------------------------------------------------
Janardhan R. Iyengar				   [email-masked]
University of Delaware		       http://www.cis.udel.edu/~iyengar
-----------------------------------------------------------------------

---

From: Randall Stewart <[email-masked]>
Date: Fri, 09 Nov 2001 12:58:11 -0600

Janardhan:

This is really a trade off question. We could have
put reassembly at the stream level or at the TSN
level. 

Doing so at the stream level pushes more overhead into
each stream (i.e. you must keep a re-assembly queue).  Also
you lose the ability to have a consistent view of stream
sequence numbers... i.e. the app can never use the SSN
for anything...

Doing so at the TSN level gives us what we have today..
only one re-assembly queue... and the app cannot use
the TSN number...

And one point to make on the comment that I just
thought of. I don't think I agree with it. I can deliver
two seperate sets of SSN's that are fragmented..

Consider

TSN=101,Flag=B,Str=1, SSN=1
TSN=102,Flag=0,Str=1, SSN=1
TSN=103,Flag=E,Str=1, SSN=1
TSN=104,Flag=B,Str=4, SSN=8
TSN=105,Flag=0,Str=4, SSN=8
TSN=106,Flag=E,Str=4, SSN=8

Now lets say 103 is lost. 

How does this loss keep me from delivery of 104-106?

If they are all on my queue I CAN recognize the B/E bits
and all pieces inbetween. I therefor CAN deliver them.

I can see where someone may implement it so you can't but
this is strickly an implementation choice to make it HOL blocking.

It is true the sender side would not be able to free up the
resoures for 104-106 (due to the reneg issue)  but that is
a different issue..

R

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Fri, 9 Nov 2001 15:37:29 -0600

Randall,

> sequence numbers... i.e. the app can never use the SSN

I think the app is fully capable of counting outgoing and
incoming records.  What would it need SSN for?

> only one re-assembly queue... and the app cannot use

What does it matter if received messages are in one
queue or another?  They do not take up more resources
by being in a different queue.

> How does this loss keep me from delivery of 104-106?

You missed the point:

  TSN=102,Flag=B,Str=4, SSN=8
  TSN=103,Flag=X,Str=1, SSN=1
  TSN=104,Flag=0,Str=4, SSN=8
  TSN=105,Flag=E,Str=4, SSN=8

Now 103 gets lost.  Stream 4 has been blocked by stream
1 until 103 is retransmitted and successfully received.
Reassembly cannot determine whether 103 contain another
interior fragment to the record be reassembled on stream
4.  If records are really large on a lot of streams, one
packet loss stands a good probability of HOL blocking all
streams.

--brian

---

From: "Armando L. Caro Jr." <[email-masked]>
Date: Fri, 9 Nov 2001 17:25:07 -0500 (EST)

> 1 until 103 is retransmitted and successfully received.

Yes, you are right... BUT, why would the sender ever transmit this
sequence. If the sending application passes down to SCTP a message which
needs fragmenting, all the fragments will be assigned TSNs in sequence.
There is no reason to interleave message fragments.

~armando

0--                                                --0
| Armando L. Caro Jr.     |       [email-masked] |
| University of Delaware  |  www.cis.udel.edu/~acaro |
0--                                                --0

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Fri, 9 Nov 2001 16:56:03 -0600

Armando,

> There is no reason to interleave message fragments.

Why would an implementation block one stream waiting for
all the fragments of another?

Randall misrepresents the socket interface: partial writes
are supported by the interface.  sendmsg can return a number
of bytes written which is less than the number requested to
be writtern.

Also, after a fork it is possible to have multiple partial
writers to a socket.  In this circumstance, the implementation
has no choice but to interleave fragments, or block all streams
waiting for one partial writer on one stream to complete (bad idea).

So, either you block on the transmit all the time (worse),
or this sequence is possible and you block at the receiver
after a lost packet.

Placing a fragment count in the DATA chunk would allow unblocked
ressembly without affecting SSN anyway.

--brian

---

From: Randall Stewart <[email-masked]>
Date: Fri, 09 Nov 2001 16:59:58 -0600

Armando:

It is also not legal..

I quote RFC2960:
"
   Fragmentation takes the following steps:

   1) The data sender MUST break the user message into a series of DATA
      chunks such that each chunk plus SCTP overhead fits into an IP
      datagram smaller than or equal to the association Path MTU.

   2) The transmitter MUST then assign, in sequence, a separate TSN to
      each of the DATA chunks in the series.  The transmitter assigns
      the same SSN to each of the DATA chunks.  If the user indicates

Stewart, et al.             Standards Track                    [Page 83]
"

It even has the MUST so there is NO wiggle room!

R

---

From: Randall Stewart <[email-masked]>
Date: Fri, 09 Nov 2001 16:58:05 -0600

> packet loss stands a good probability of HOL blocking all

You missed a rule... what you show above is not legal.

TSN's on fragmented chunks MUST be assigned in sequence
so you cannot have the intervening chunk 103...

R

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Fri, 9 Nov 2001 18:19:59 -0600

Randall,

This is a requirement on the SCTP implementation fragmenting
chunks, not on the ULP.  The ULP can do partial writes.  I
don't see where it says that the sender must wait for all
partial writes before sending the first fragment.

--brian

---

From: Randall Stewart <[email-masked]>
Date: Sat, 10 Nov 2001 05:28:34 -0600

> > > >   TSN=105,Flag=E,Str=4, SSN=8

I am refering to your example above. Notice that
TSN 103 is in the MIDDLE of the fragmented message
on stream 4 SSN 8. 

This is not a legal thing..

And as to your ULP doing partial writes... each sendmsg() is
considered a message... so I am unsure how you can get a partial
write....

My comment though, was on the above.. the example can NOT happen
unless the SCTP implementation violates the protocol.

R

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Sat, 10 Nov 2001 06:06:24 -0600

Randall,

> considered a message... so I am unsure how you can get a partial

sendmsg() is permitted to return without writing all of the
bytes supplied (partial write).  Also, if MSG_EOR is not set,
a subsequent sendmsg() is for the same record.  Both these
are partial writes.  If you are trying to say (which the RFC
does not) that all writes must be complete or fail they you
cannot have a TCP-like socket interface (only a UDP-like one).
Then SCTP would have a serious deficiency over TCP (which can
do partial writes on multiple forked processes or threads).

Which defficiency do you prefer?

--brian

---

From: Randall Stewart <[email-masked]>
Date: Sat, 10 Nov 2001 08:55:13 -0600

> Which defficiency do you prefer?

It really does not matter if the interface supports partial
write or not. If you write a huge message (where a
partial write would be) then guess what. The sending
SCTP stack CANNOT start to fragment the message and assign
TSN's until it has the complete message. Either that
or block other streams from assignment of TSNs until you get
the complete message and all of the partial messages sequential 
TSN's have been assigned... This IS REQUIRED by RFC2960. Implementations
choice, either support a more UDPish model or hold other
streams while collecting a big message to send. 

And has to TCP defficiency.. I don't see it that way.. SCTP
is message oriented NOT a byte stream. So there are going
to be some places where SCTP cannot map totally to TCP.. thats
ok.. its what message oriented means :>

And to me UDP has always been the better model for SCTP NOT
TCP ... thats why the socket-api HAS a UDP model.. oh and
our KAME SCTP that will be out soon, on its first release,
will NOT support the TCP model.. When it does I will require
the whole message in one write.. like UDP since this IS
the better model :>>

R

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Sat, 10 Nov 2001 19:04:25 -0600

Randall,

> streams while collecting a big message to send. 

Please cite where in RFC2960 it states that the SCTP sender
must wait for a complete message from ULP before if can
send the first fragment.

> ok.. its what message oriented means :>

Well, if the SCTP must wait for the entire message before sending,
this would be a reason:

	a)  To not use SCTP.
	b)  To not use SCTP even for a single stream.

> the whole message in one write.. like UDP since this IS

Everyone knows that full message writes are a deficiency
of the UDP model.  One that it seems have been carried into
SCTP.

As SCTP's objective was to address some deficiencies of 
TCP, I would say that SCTP has failed dismally in that
regard, particularly as it cannot support decent latency
with partial writes.

--brian

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Sat, 10 Nov 2001 20:46:44 -0600

Randall,

On Sat, 10 Nov 2001, Randall Stewart wrote:

> TSN's until it has the complete message.

XNS 5.2 sendmsg:

   RETURN VALUE

    Upon successful completion, sendmsg() function returns the number
    of bytes sent.  Otherwise, -1 is returned an errno is set to
    indicate the error.

Seems to me as though SCTP sendmsg must always return 0 or -1
unless MSG_EOR is set and there is sufficient room to buffer
the entire message in the send buffer.

Therefore, if this restriction really applies to SCTP, then
SCTP cannot support partial write on the socket interface at all.
This is a serious deficiency.

--brian

> Either that

---

From: Randall Stewart <[email-masked]>
Date: Sun, 11 Nov 2001 05:54:27 -0600

> must wait for a complete message from ULP before if can

Please go back and read what I cited before, and maybe
go back and read carefully what I am saying.. you obviously
are not... since you are mis-interpreting what I said.

>         b)  To not use SCTP even for a single stream.

You reach interesting conclusions.. the same logic (which
I don't grasp by the way) would apply to UDP then... strange
it does seem to be popular :-)

> of the UDP model.  One that it seems have been carried into

Gee, imagine that, I have worked with UDP for close to 12
years and never noted what "everyone" knows. And even
more surprising.. UDP is one of the most popular
choices with IP.. hmm.. I wonder who everyone is..
besides Brian...

> regard, particularly as it cannot support decent latency

You have been anti-IP and anti-SCTP since as far back
as I can remember. So none of your comments surprise
me.

R

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Sun, 11 Nov 2001 06:23:20 -0600

Randall,

> are not... since you are mis-interpreting what I said.

The section you cited said that SCTP sender must assign
sequential TSN to messages it fragments as received from
the user.  It does not say that message must be received
complete from the user.

> it does seem to be popular :-)

SCTP does not improve on UDP.  In this regard is does not
even meet TCP's abilities.  TCP is rather popular too.

> choices with IP.. hmm.. I wonder who everyone is..

Every application that uses TCP.

> as I can remember. So none of your comments surprise

You mistake me.  I'm not anti-SCTP, I'm anti-Randall. ;)

There is no reason why SCTP should not be permitted to
perform partial-writes and partial-sends within a stream
as does TCP.  Even considering the sorry design of
fragmentation.  I don't see that even the MUST that you
cite is justifed.

As a result of this deficiency (which was not made apparent
in the past), the SIGTRAN UAs (M3UA/SUA) that relied on
what we thought were SCTP's ability to segment and reassemble
message without latency impacts, will have to perform segmentation
and reassembly at the ULP-level.

So, tell me:  what is the use of SCTP's fragmentation and
reassembly abilities?

Also, the AS should clearly state that SCTP has this weakness
in comparison to TCP: that applications requiring minimal
latency (gee, signalling applications) should consider either
making messages sufficiently small that SCTP does not fragment,
or should consider not using SCTP.

--brian

---

From: Randall Stewart <[email-masked]>
Date: Sun, 11 Nov 2001 06:19:21 -0600

Brian:

Gee a quote from a future informational RFC.. how is this
preventing someone from implementation of something
different... Go read the re-read the thread and
look at the statements I made. If you take the time to
attempt to understand them you may figure out alternatives...

And of course since the socket api is informational it
has no binding effect..

Regards
R

---

From: Randall Stewart <[email-masked]>
Date: Sun, 11 Nov 2001 06:23:49 -0600

You have been anti-IP and anti-SCTP since as far back
as I can remember. So none of your comments surprise
me, no matter how mis-guided and wrong they are.

R

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Sun, 11 Nov 2001 06:30:24 -0600

Randall,

I was quoting a standard, not a future informational rfc.

--brian

---

From: Randall Stewart <[email-masked]>
Date: Sun, 11 Nov 2001 06:37:52 -0600

> the user.  It does not say that message must be received

You are finally bothering to read what I wrote.. or catching
up and figuring it out.. This is the point I have been making
to you all along.

> even meet TCP's abilities.  TCP is rather popular too.

This is your opinion.

> Every application that uses TCP.

So you speak for every application now.. interesting ..

> fragmentation.  I don't see that even the MUST that you

This is one thing the WG designed, not I. The original
design in MDTP was different. And again, I never said
SCTP could NOT support partial writes.. you said that
not I.

> So, tell me:  what is the use of SCTP's fragmentation and

I think you are just unreasonable. Fragmentation will work
for the sigtran UA's. 

> or should consider not using SCTP.

And signalling has such huge messages dont they.. all the
time when I worked in Nynex I saw all kinds of IAM messages
that must have been the huge size of 220 bytes.. maybe..

Gee. If I remember right fragmentation was not even wanted
by Sigtran originally.. thats why they took OUT the design
that was originally in it.

R

---

From: Randall Stewart <[email-masked]>
Date: Sun, 11 Nov 2001 11:24:27 -0600

> I was quoting a standard, not a future informational rfc.

My mistake... it sounds very similar to what is in the
sockets draft... 

Hmm.. I think the fundamental issue is that in TCP's
case there never is a end of record since you have
a stream of bytes. In UDP a send seems always a message
to send in a IP datagram.. 

I do see that the man page for send states that a MSG_EOR
can be applied to indicate an end of record.. but I don't
think in UDP's case that this works.. unless it is in
the sockets layer above the UDP module. I have looked in
the UDP side and it just gets a chain of mbufs that it considers
a record... so unless the base socket's layer supports
the use of MSG_EOR then I doubt it is supported in
any of the IP protocols (at least in the BSD stack). Of
course different stacks may do different things... some
may not even have a sockets layer.. which in that
case they can support anything they want :>

R

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Sun, 11 Nov 2001 15:37:36 -0600

Randall,

> case they can support anything they want :>

It is supported by sockets for ISO protocols; but, obviously
you have no familiarity with them.   MSG_EOR is used for
record delimitation with partial write.  Regardless, sendmsg
must return the number of bytes "sent" not the number of bytes
stored to be sent later.

So you are wrong.  The SCTP implementation must return 0 or -1
on partial write because it has not sent anything.  This
means that it cannot store messages for later delivery and the
only successful call to sendmsg must be one with a complete
record.

TCP does not have this deficiency.

--brian

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Sun, 11 Nov 2001 16:06:12 -0600

Randall,

> that must have been the huge size of 220 bytes.. maybe..

Randall, SCCP CO supports byte-streams without end.  TCAP
messages can be very large (contain complete SMS messages
or voice-mail records).  Take the time to learn what you
are talking about.

> that was originally in it.

Fragmentation and reassembly are a part of SCCP.  SUA assumed
that SCTP would do this, but now SUA must do it for SCTP because
is it so lacking an has missed this essential feature.

One stream blocking others from the cwnd because the entire
message must have sequential TSNs is unacceptable from a
performance standpoint also.

--brian

---

From: Randall Stewart <[email-masked]>
Date: Sun, 11 Nov 2001 16:27:48 -0600

> must return the number of bytes "sent" not the number of bytes
Well that is not what I have found in the sockets layer. I
did a little peak into uipc_socket.c in BSD. There is
a flag that can set by the protocol that will enable
the use of the MSG_EOR flag. If the socket itself has
the flag PR_IMPLOPCL, then the protocol stack implementation
will be given:

PRUS_MORETOCOME or PRUS_EOF depending on the
setting of the MSG_EOR flag.

> only successful call to sendmsg must be one with a complete

If that were true then no matter if you implement a ISO stack
or a IP stack BSD sockets would not support it. Please
download the FreeBSD implementation and take a look at
uipc_socket.c about line 640 or so...

The pru_send() call will return either 0 or a postive value
indicating the error. All of the send/sendmsg/sendto calls
boil down into this one call. These are API issues though
and not really standardized in the IETF.

> TCP does not have this deficiency.

TCP is a byte stream protocol. By definition there is NO
message boundary's. This is not a deficiency.. it is just
the way the two protocols were designed differently.

R

---

From: Randall Stewart <[email-masked]>
Date: Sun, 11 Nov 2001 16:30:18 -0600

> or voice-mail records).  Take the time to learn what you

Note, I said signalling messages. TCAP and company are
a bit of a different critter that I have not ever
worked with. SMS messages do have limits in sizes though, at
least my GSM phone does not send more than a limited
number of bytes at a time.

> performance standpoint also.

This may be your design but it would not be mine. The implementation
we put out will block only the sending stream and wait to
collect the message.

R

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Sun, 11 Nov 2001 17:41:25 -0600

Randall,

> we put out will block only the sending stream and wait to

This is unacceptable to SUA as it may need to wait indefinitely
for the final segment of a message from the SS7 network.  Therefore,
partial writes to SCTP cannot be supported unless SCTP sends partial
records and does not block in this fashion.

SCTP fragmentation unusable in this scenario.  SUA will need
to fragment for SCTP (keep records down below a path MTU) to
avoid this unbounded latency.  For M3UA the impact is high.  Any
gains acheived by placing separare SLS on streams will be lost
if large SIF max interwork into smaller SCTP MTUs as streams
will become blocked both at transmit and by the congestion window,
*always*, even if no packet is lost.

Thus the proposal to SIGTRAN.

--brian

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Sun, 11 Nov 2001 18:18:07 -0600

Randall,

> uipc_socket.c about line 640 or so...

I was not aware that FreeBSD even attempted XOpen/XNS 5.2
or POSIX compliance.

> and not really standardized in the IETF.

But are standardized elsewhere.  SCTP's "feature" cannot
support partial write on the standard interface.

> the way the two protocols were designed differently.

TCP supports the concept of messages boundaries: it is just
that the application must place its own delimiters.  The
POSIX, XOpen/XPG XNS 5.2, Unistd, .. interface supports
one process performing a partial write (and its segment is
actually sent, thus sendmsg() returns >1) and other processes
are blocked from sending segments until the partial writer
completes its write.  This blocking is on a *single* byte
stream and is not *between* streams as in the case of SCTP
as you describe it.

If one permitted partial writes to SCTP, there is a deadlock
scenario where the remaining send buffer is not large enough
to hold the entire message.  There also exists a choking
scenario where the remaining rwnd or crwnd is not sufficiently
large enough to hold the entire message and other streams are
blocked off of the rwnd or crwnd until all of the fragments can
be transmitted and acknowledged.  Thus one big message can
block many streams with little messages, degrading their
performance.

The necessity to place all of the fragments together and
contiguous in TSN give large messages priority over smaller
messages in the use of the send buffer, rwnd, and crwnd.
What is worse is that the delay experienced by smaller messages
on other streams is degraded disproprortional to the bandwidth
and latency of the underlying network: these smaller messages
are discriminated against by this contiguous TSN policy
applied to larger messages that gives them preference over
(blocks) smaller messages on other streams.

The mechanism that can be applied by the ULP to ensure that
these non-linear queueing effects do not impact the performance
and latency of smaller message streams alongside larger message
streams is to segment all larger messages beneath a path MTU
before delivery to SCTP.  Then SCTP will not have an opportunity
to apply this unfair policy.

That is what is proposed to SIGTRAN.

--brian

---

From: Randall Stewart <[email-masked]>
Date: Sun, 11 Nov 2001 20:25:54 -0600

> avoid this unbounded latency.  For M3UA the impact is high.  Any

Seems to me one would not need to keep things down at a MTU level
but instead only around a fragmentation size of say 32-64k.. which
is a reasonable point where you will not be blocked. One would
want to look at the socket API min's and pick a safe boundary
but I would think no smaller than 32k.. optimize the nominal
case and you only end up fragmenting when you hit the 
less common case..

> *always*, even if no packet is lost.

Well, since the fragmentation happens at the TSN level I
don't see anyway around this issue... one can easily imagine
a 32k message not having any trouble with this.. but not
so with a  100k message.. for example..

R

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Sun, 11 Nov 2001 21:11:03 -0600

Randall,

> case and you only end up fragmenting when you hit the 

Messages above an MTU still block the access of other streams
to the remaining rwnd and cwnd.

> so with a  100k message.. for example..

A message of two MTUs (two fragments) can block other stream
access to a current rwnd or cwnd < 2MTU.  Therefore, this
problem can occur at any message size above an MTU.

--brian

---

From: Qiaobing Xie <[email-masked]>
Date: Mon, 12 Nov 2001 12:41:30 -0600

Brian,

I read through some of your emails in this thread (but not all of them).
From what I've seen, your problem seems to be the approach you insist on
using when you design your applications (e.g., the upper layers of your
vision). Unlike most people who normally architect their application
design carefully in such a way so as to take the best advantage of
services provided by the lower layer(s), you seem to favor designs that
fit in the worst possible way to whatever services available from the
lower layers and then insist on the lower layers being changed to make
your favored design to work. This approach is very strange to me.

-Qiaobing

---

From: Laurent Glaude <[email-masked]>
Date: Mon, 12 Nov 2001 13:53:07 -0500

Hi all,

I have just spent the last 1 1/2 hours trying to understand the technical
issues amidst the flames and smoke,
and I'd like to make a comparison to the usage of streams in SCTP and
another field of study in computer science. Please correct me if I'm wrong.

It seems to me that we can correlate the multiple streams, sharing
congestion and receive windows over an
association as to processes sharing CPU cycles on a system. Namely, if the
size of a message sent on a stream is seen as the process execution time,
and we send, on the following streams, say, the following
messages:

stream 0 : msg 1 ( 200 bytes)
stream 1 : msg 2 ( 100000 bytes)
stream 3 : msg 3 ( 1000 bytes )
stream 4 : msg 4 ( 10000 bytes )

Then maybe what we want to avoid in this case, is having message # 2
"starving" the other streams while
it completes, and so on? Not that I want to add scheduling here, but I
would like to make the following point:

A lot of the impact of streams  regarding head-of-line blocking in SCTP is
related to the distribution of the messages across the streams, and the
mean / variance of message size. A few cases:

A - With small messages, and a small variance of the message size ( as in
the case of MSUs, say from 100 to 250 bytes )a uniform distribution
achieves the best results, provided there are a fair amount of streams
used. An example where the head-of-line blocking is greatly reduced is when
sending 200 byte messages uniformly distributed across, say, 40 streams
( assuming 1500 byte MTU ). However, multiplexing 200 byte messages
uniformly over 4 streams gives no advantage whatsoever,  (again using a
1500 byte MTU), because chances are that there is a data chunk for every
stream in every packet.

B - The more biased a distribution becomes,  the less likely to avoid
head-of-line blocking. As an example,
it doesn't matter if one uses 1000 streams, if 80 % of the messages are
sent to stream zero.

C - As discussed previously, a high variance of the message size does not
cause head-of-line blocking
per se, but it can starve the other streams momentarily. Or actually a high
message size, for that matter,
say if sending only 1 megabyte messages, but that's extreme.

D - Fortunately, for some applications, it is possible to predict the
distribution of streams and the mean
/ variance of message size.

Just my .02

-Laurent

--
Laurent Glaude
Software Engineer
Signaling Systems Group, Performance Technologies
http://www.pt.com

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Mon, 12 Nov 2001 14:24:06 -0600

Qiaobing,

Please explain.

--brian

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Mon, 12 Nov 2001 14:29:19 -0600

Laurent,

Message 2 would not starve stream 3 and 4 if there were separate
associations for each stream.  Just another reason not to mux.

--brian

---

From: "Denio Mariz" <[email-masked]>
Date: Tue, 13 Nov 2001 12:18:25 -0200

Hello Brian,

> associations for each stream.  Just another reason not to mux.

I cannot see how SCTP could have separate associations for each stream.
Following RFC 2960 (page 13), "Two SCTP endpoints MUST NOT have
more than one SCTP association between them at any given time."

There is another document posing a diferent behavior ?
Thank you.

Denio.

---

From: Jacob Heitz <[email-masked]>
Date: Tue, 13 Nov 2001 07:29:34 -0800

An endpoint is not a host. It is simply the endpoint of an
SCTP association, purely a logical thing.
A host may have multiple endpoints.

---

From: Laurent Glaude <[email-masked]>
Date: Tue, 13 Nov 2001 12:38:00 -0500

Yes, message 2 would not starve.

However, I am skeptic over the approach of using one association per stream,
as I am wary of the extra burden it can add to a system, and the complexity it
can add to a user application. If possible, I would prefer to find a solution
within SCTP's use of streams rather than using multiple associations.
A few comments:

1 - The number of streams is bound to be large in order to effectively
reduce head-of-line blocking. Thus, so would be the number of associations
with this approach, although some kind of "association pool" could reduce
the number of associations needed.

2 - I also see issues in the replication of structures and services offered
by SCTP. As an example, X associations also mean X congestion windows for
every IP addresses of the peer endpoint. (To the *same* endpoint, I
might add). I am wary here, especially since X is bound to be large. Another
example is the use of heartbeats. It would not be very efficient to let
every association send heartbeats, but on the other side, controlling
heartbeats at the application just adds a more complexity (again).

3 - The suggestion of using one association for each stream does bring
an interesting point. In this case, by the TCP fairness principle, all
"streams" (associations) compete equally for access to the bandwith.

I don't know for sure, but perhaps this is what we should strive for SCTP
streams. I do have a few ideas on how to do it (see #4), but I need to ponder
more...

4 - Or, alternatively, can't this fainess be achieved using the stream-based
flow control extension?

-Laurent

---

From: Qiaobing Xie <[email-masked]>
Date: Tue, 13 Nov 2001 12:53:53 -0600

Laurent: 

I don't know whether this comment is general or specific to Sigtran xUA
designs. If former, this is about system architecture and thus makes
little sense if we don't know what kind of applications are concerned
here. If latter, then I'd say that "using one association per stream" is
a very bad design decision.

If inter-stream head-of-line (HOL) blocking is the problem this
discussion is trying to solve, then this is not the solution. In fact, I
am not even sure that there is a valid problem in the first place :-)

Say, your design calls for 10k streams between two communication apps
and to avoid HOL blocking you create 10K associations (one per stream).
Then I'd imagine that inside your application you will have to use some
API (say, a socket API) to manage those 10k associations. This means
that you probably will need some kind of internal dispatching function
to serve the read/write events of those 10k file descriptors. Guess
what, this dispatching function becomes your new inter-stream HOL
blocking focal point; if one of the fd grabs the service of the
dispatcher the other fds (i.e. other streams) will have to wait for
their turn to be served. What you have achieved here is no more than
just pushing the HOL blocking point around.

-Qiaobing