Subject: [Tsvwg] SCTP_MAXSEG socket option semantics

From: Mark Butler <[email-masked]>
Date: Mon, 26 Dec 2005 14:58:07 -0700

Hello all,

The current sctpsocket draft specifies SCTP_MAXSEG as specifying the 
maximum data payload to put into any outgoing data chunk, or the maximum 
fragmentation size.  The first problem with this definition is that it 
is incomplete - if only the fragmentation size is adjusted, the packet 
bundling process may reassemble multiple fragments of the same message 
back into the same outgoing packet, which is surely not desirable behavior.

The presumed purpose of such an option is to reduce total packet size.  
Messages larges than SCTP_MAXSEG are not rejected - they are fragmented. 
But even in the absence of large messages, a series of small messages 
may add up to nearly the PMTU size, making the option relatively 
ineffective.

So I suggest that the definition of SCTP_MAXSEG be changed to both bound 
the data fragmentation point and to advise the packet bundler to switch 
to a new packet after the sum of the chunk lengths meets or exceeds 
MAXSEG + 20, where 20 is the data chunk header size.  This definition 
would be compatible with the TCP_MAXSEG option and the advisory MSS 
information from the route tables on many systems.

Comments anyone?

 - Mark

_______________________________________________
tsvwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tsvwg

---

From: Michael Tuexen <[email-masked]>
Date: Mon, 26 Dec 2005 23:19:06 +0100

Hi Mark,

see my comments below.

Best regards
Michael

On Dec 26, 2005, at 10:58 PM, Mark Butler wrote:

> Hello all,
>
> The current sctpsocket draft specifies SCTP_MAXSEG as specifying  
> the maximum data payload to put into any outgoing data chunk, or  
> the maximum fragmentation size.  The first problem with this  
> definition is that it is incomplete - if only the fragmentation  
> size is adjusted, the packet bundling process may reassemble  
> multiple fragments of the same message back into the same outgoing  
> packet, which is surely not desirable behavior.
As far as I understand the option, it defines the maximum payload to  
put in a chunk. It is OK to
bundle multiple DATA chunks in one packet. This might be useful for  
example if the MTU shrinks. You can
than still continue without IP fragmentation.
>
> The presumed purpose of such an option is to reduce total packet  
> size.  Messages larges than SCTP_MAXSEG are not rejected - they are  
> fragmented. But even in the absence of large messages, a series of  
> small messages may add up to nearly the PMTU size, making the  
> option relatively ineffective.
>
> So I suggest that the definition of SCTP_MAXSEG be changed to both  
> bound the data fragmentation point and to advise the packet bundler  
> to switch to a new packet after the sum of the chunk lengths meets  
> or exceeds MAXSEG + 20, where 20 is the data chunk header size.   
> This definition would be compatible with the TCP_MAXSEG option and  
> the advisory MSS information from the route tables on many systems.
If you want to disallow bundling, you might want to have an  
additional socket option which disables bundling.

---

From: Mark Butler <[email-masked]>
Date: Mon, 26 Dec 2005 17:09:24 -0700

Hello Michael - comments below.

>> maximum data payload to put into any outgoing data chunk, or  the 
>> maximum fragmentation size.  The first problem with this  definition 
>> is that it is incomplete - if only the fragmentation  size is 
>> adjusted, the packet bundling process may reassemble  multiple 
>> fragments of the same message back into the same outgoing  packet, 
>> which is surely not desirable behavior.
>

Yes that is what the documentation says.  But if you have a 1500 byte 
MTU, a 300 byte SCTP_MAXSEG, and then send a 1000 byte message - what 
happens?  As currently defined the 1000 byte message is broken up into 
four fragments, and then bundled back together in the same packet 
(assuming the implementation does transmit bundling).  That just wastes 
60 bytes of chunk headers and a bunch of CPU overhead.  What purpose has 
been served?

>> small messages may add up to nearly the PMTU size, making the  option 
>> relatively ineffective.
>>

It is not very helpful to turn it off completely.  I am just suggesting 
that the only useful implementation of SCTP_MAXSEG quits bundling when 
the packet is above the specified size (roughly speaking).  Thanks for 
your comments.

 - Mark

---

From: Michael Tuexen <[email-masked]>
Date: Tue, 27 Dec 2005 01:19:07 +0100

Hi Mark,

see my comments in-line.

Best regards
Michael

On Dec 27, 2005, at 1:09 AM, Mark Butler wrote:

>>> multiple fragments of the same message back into the same  
>>> outgoing  packet, which is surely not desirable behavior.
>>
>> As far as I understand the option, it defines the maximum payload  
>> to  put in a chunk. It is OK to
>> bundle multiple DATA chunks in one packet. This might be useful  
>> for  example if the MTU shrinks. You can
> Yes that is what the documentation says.  But if you have a 1500  
> byte MTU, a 300 byte SCTP_MAXSEG, and then send a 1000 byte message  
> - what happens?  As currently defined the 1000 byte message is  
> broken up into four fragments, and then bundled back together in  
> the same packet (assuming the implementation does transmit  
> bundling).  That just wastes 60 bytes of chunk headers and a bunch  
> of CPU overhead.  What purpose has been served?
Well, as I said: If the MTU drops to let us say 500 byte, you can  
retransmit the chunks in four packets
without using IP fragmentation.
I do not this that something is 'wasted'. The user requested it.
>>> size.  Messages larges than SCTP_MAXSEG are not rejected - they  
>>> are  fragmented. But even in the absence of large messages, a  
>>> series of  small messages may add up to nearly the PMTU size,  
>>> making the  option relatively ineffective.
>>>
>>> So I suggest that the definition of SCTP_MAXSEG be changed to  
>>> both  bound the data fragmentation point and to advise the packet  
>>> bundler  to switch to a new packet after the sum of the chunk  
>>> lengths meets  or exceeds MAXSEG + 20, where 20 is the data chunk  
>>> header size.   This definition would be compatible with the  
>>> TCP_MAXSEG option and  the advisory MSS information from the  
>>> route tables on many systems.
>>
> It is not very helpful to turn it off completely.  I am just  
> suggesting that the only useful implementation of SCTP_MAXSEG quits  
> bundling when the packet is above the specified size (roughly  
> speaking).  Thanks for your comments.
But you could get your behaviour by combining the two socket options.

---

From: Mark Butler <[email-masked]>
Date: Mon, 26 Dec 2005 21:03:54 -0700

Michael Tuexen wrote:

>> broken up into four fragments, and then bundled back together in  the 
>> same packet (assuming the implementation does transmit  bundling).  
>> That just wastes 60 bytes of chunk headers and a bunch  of CPU 
>> overhead.  What purpose has been served?
>

Yes - however the differential cost is heavy and the differential 
benefit is rare.  Also, If the option is not going to behave like 
TCP_MAXSEG, it should be called something else.  Otherwise it is 
misleading.  And then when somebody wants a TCP_MAXSEG like option they 
have to call it something like SCTP_SHOULD_HAVE_BEEN_MAXSEG.

Extending the semantics would preserve backward compatibility - that of 
guaranteeing chunk payloads are no larger than the specified size.  So 
why not enhance compatibility and solve the general packet size problem 
as well?  - something that has common and well understood benefits in 
many environments, rather than something that incurs ongoing costs with 
much reduced benefit.

Say for example that I am on a local net with 9000 byte jumbo frames and 
PMTU discovery enabled.  I set SCTP_MAXSEG to 1400, connect to a remote 
SCTP host on a traditional Ethernet network and send a 64 KB message, 
and PMTU. So a couple of ~ 8500 byte packets go out with six fragments 
from the same message each with the DF bit set.  When the first packet 
reaches the remote gateway router, it will be dropped and an ICMP 
datagram too big message will be returned, and eventually the source 
stack will discover the PMTU and send the right size packets.  Of 
course, the chunks (all 46 of them) are the right size to go through, 
after this multiple RTT process has completed, the retransmit timer has 
expired, etc.

At the same time, a similar TCP connection is established, same network 
setup, PMTU discovery enabled, TCP_MAXSEG set to 1400, a 64KB message..  
The TCP stack breaks up the 64KB write into 46 packets with the DF bit 
set, sends them as the congestion window allows, and all of them arrive 
without an expensive PMTU discovery process.

Likewise, if SCTP_MAXSEG had similar extended semantics, all the chunks 
would have payloads less than 1400 bytes (as promised today), and the 
packets would all go through without PMTU discovery, just like TCP.

 - Mark

---

From: Randall Stewart <[email-masked]>
Date: Tue, 27 Dec 2005 07:17:37 -0500

> than still continue without IP fragmentation.

Yep.. this is its purpose ... I believe the
RDDP folks expressed a desire to control this for
PMTU changes...

>> to switch to a new packet after the sum of the chunk lengths meets  or 
>> exceeds MAXSEG + 20, where 20 is the data chunk header size.   This 
>> definition would be compatible with the TCP_MAXSEG option and  the 
>> advisory MSS information from the route tables on many systems.
> 
> If you want to disallow bundling, you might want to have an  additional 
> socket option which disables bundling.

This already exists .. its defined as follows:
"
7.1.12.  Enable/Disable message fragmentation (SCTP_DISABLE_FRAGMENTS)

    This option is a on/off flag and is passed an integer where a non-
    zero is on and a zero is off.  If enabled no SCTP message
    fragmentation will be performed.  Instead if a message being sent
    exceeds the current PMTU size, the message will NOT be sent and
    instead a error will be indicated to the user.
"

R

-- 
Randall Stewart
803-345-0369 <or> 815-342-5222(cell)

---

From: Sridhar Samudrala <[email-masked]>
Date: Tue, 27 Dec 2005 16:14:57 -0800

> This already exists .. its defined as follows:

I think what Mark is suggesting is a SCTP_DISABLE_BUNDLING option and
would like this to be on if SCTP_MAXSEG is set.

Currently SCTP_MAXSEG is defined as the maximum size of a SCTP data chunk
payload. Instead he seems to be suggesting this to be treated as the maximum
size of SCTP packet payload.
I think both the definitions may be useful in different scenarios. So i
would suggest adding a new option SCTP_DISABLE_BUNDLING and leave the
definition of SCTP_MAXSEG as it is now.

Setting SCTP_MAXSEG and enabling SCTP_DISABLE_BUNDLING should give what
Mark wants.

Thanks
Sridhar

---

From: Mark Butler <[email-masked]>
Date: Thu, 29 Dec 2005 15:56:09 -0700

Hello Randall, Comments inline...

> RDDP folks expressed a desire to control this for

The RDDP SCTP draft says that fragmentation at the SCTP layer should be 
avoided.  However the SCTP_MAXSEG option cannot be used to get _less_ 
SCTP layer fragmentation, only _more_ SCTP layer fragmentation.  What an 
RDDP SCTP implementation needs is a way to determine the current maximum 
data chunk size that will not get fragmented.  This can be done with a 
getsockopt(...,SCTP_STATUS,...) call.  If the RDDP layer has better 
knowledge of the true PMTU, it can reduce the message size by itself.

SCTP needs a packet level MSS, like TCP.  A typical reason to clamp the 
MSS is to prevent PMTU discovery delays when you know that the PMTU is 
lower (or likely to be lower) than first hop MTU.  Another common 
application is limiting queuing delays on transit routers.  SCTP_MAXSEG 
doesn't currently help here, because it doesn't limit the packet size, 
only the fragment size. The primary motivation for SCTP is reduced 
message delivery latency relative to TCP,  correct? 

I can see the utility of two independent options.  In the name of not 
maintaining a radically different semantic for "Maximum Segment Size" 
than TCP and other protocols (spread throughout routing utilities, 
advisory routing metrics, and network documentation everywhere) I 
recommend that we make SCTP_MAXSEG behave like TCP_MAXSEG, and adopt a 
compatible definition of the SCTP MSS.

Then we should rename the option to clamp the data fragmentation point. 
Perhaps SCTP_MAX_DATA_SEG.  This is pretty obscure - the only 
differential use case is to avoid temporary IP layer fragmentation after 
PMTU discovery without the minimal additional overhead of reducing the 
packet size to match. 

MSS negotiation like TCP would also help.  This would require an INIT / 
INIT ACK parameter that advertised the local MSS.  On reception, each 
peer would then normally reduce its sending MSS accordingly. And in this 
context I mean a packet size based MSS like what TCP has: MSS  = PMTU - 
mandatory headers.

(more below)

>    instead a error will be indicated to the user.

An option to disable bundling is different from an option to disable 
fragmentation.  You can certainly still fragment if you do not bundle.  
If one does not want bundling, the best way to disable it for practical 
purposes is to set SCTP_NODELAY. 

- Mark

---

From: Michael Tuexen <[email-masked]>
Date: Fri, 30 Dec 2005 00:54:38 +0100

On Dec 29, 2005, at 11:56 PM, Mark Butler wrote:

>>>> The current sctpsocket draft specifies SCTP_MAXSEG as  
>>>> specifying  the maximum data payload to put into any outgoing  
>>>> data chunk, or  the maximum fragmentation size.  The first  
>>>> problem with this  definition is that it is incomplete - if only  
>>>> the fragmentation  size is adjusted, the packet bundling process  
>>>> may reassemble  multiple fragments of the same message back into  
>>>> the same outgoing  packet, which is surely not desirable behavior.
>>>
>>>
> The RDDP SCTP draft says that fragmentation at the SCTP layer  
> should be avoided.  However the SCTP_MAXSEG option cannot be used  
> to get _less_ SCTP layer fragmentation, only _more_ SCTP layer  
> fragmentation.  What an RDDP SCTP implementation needs is a way to  
> determine the current maximum data chunk size that will not get  
> fragmented.  This can be done with a getsockopt 
> (...,SCTP_STATUS,...) call.  If the RDDP layer has better knowledge  
> of the true PMTU, it can reduce the message size by itself.
>
> SCTP needs a packet level MSS, like TCP.  A typical reason to clamp  
> the MSS is to prevent PMTU discovery delays when you know that the  
> PMTU is lower (or likely to be lower) than first hop MTU.  Another  
> common application is limiting queuing delays on transit routers.   
> SCTP_MAXSEG doesn't currently help here, because it doesn't limit  
> the packet size, only the fragment size. The primary motivation for  
> SCTP is reduced message delivery latency relative to TCP,  correct?
> I can see the utility of two independent options.  In the name of  
> not maintaining a radically different semantic for "Maximum Segment  
> Size" than TCP and other protocols (spread throughout routing  
> utilities, advisory routing metrics, and network documentation  
> everywhere) I recommend that we make SCTP_MAXSEG behave like  
> TCP_MAXSEG, and adopt a compatible definition of the SCTP MSS.
So what is the difference between your SCTP_MAXSEG and setting a MTU  
per assoc/path? Setting the MTU
is already possible.
>
> Then we should rename the option to clamp the data fragmentation  
> point. Perhaps SCTP_MAX_DATA_SEG.  This is pretty obscure - the  
> only differential use case is to avoid temporary IP layer  
> fragmentation after PMTU discovery without the minimal additional  
> overhead of reducing the packet size to match.
> MSS negotiation like TCP would also help.  This would require an  
> INIT / INIT ACK parameter that advertised the local MSS.  On  
> reception, each peer would then normally reduce its sending MSS  
> accordingly. And in this context I mean a packet size based MSS  
> like what TCP has: MSS  = PMTU - mandatory headers.
I'm not sure what you mean here. TCP has a header of limited size.  
SCTP has control chunks and there
is no limit. Even if you limit the size of DATA chunks, SCTP packets  
can be larger. So if you want small
packets, limit the (path) MTU. If you want to be able to retransmit  
without using IP fragmentation in
case of an MTU clamp, lower the fragmentation point.

>>>> both  bound the data fragmentation point and to advise the  
>>>> packet bundler  to switch to a new packet after the sum of the  
>>>> chunk lengths meets  or exceeds MAXSEG + 20, where 20 is the  
>>>> data chunk header size.   This definition would be compatible  
>>>> with the TCP_MAXSEG option and  the advisory MSS information  
>>>> from the route tables on many systems.
>>>
>>>
> An option to disable bundling is different from an option to  
> disable fragmentation.  You can certainly still fragment if you do  
> not bundle.  If one does not want bundling, the best way to disable  
> it for practical purposes is to set SCTP_NODELAY.
> - Mark
>
>
>
>

---

From: Mark Butler <[email-masked]>
Date: Thu, 29 Dec 2005 18:35:41 -0700

> per assoc/path? Setting the MTU

One problem is that (from what I can tell) the RFC2960 standard MTU is 
the IP layer MTU. Am I wrong?  Having the user specify the MTU in IP 
layer terms is a bit of a layering violation - what if you have IPIP or 
an IPSEC tunnel - should the MTU be the inner MTU or the outer MTU? 

Say one wants to add TCP like MSS negotiation to SCTP .  Each end may 
have a different encapsulation - there might be an IPv4 / IPv6 NAT in 
between, for example. For negotiation to work properly, each end needs 
to send something that is independent of the IP header size.  Otherwise 
if an IPv6 host talks to an IPv4 host, one will undershoot and the other 
will overshoot the proper size.   So unless the SCTP MTU should be 
interpreted at the SCTP layer instead of the IP layer, there needs to be 
an MSS or something like it.

>> point. Perhaps SCTP_MAX_DATA_SEG.  This is pretty obscure - the  only 
>> differential use case is to avoid temporary IP layer  fragmentation 
>> after PMTU discovery without the minimal additional  overhead of 
>> reducing the packet size to match.
>> accordingly. And in this context I mean a packet size based MSS  like 
>> what TCP has: MSS  = PMTU - mandatory headers.
>

There is a 12 byte mandatory header with SCTP.   TCP defines the MSS for 
negotiation purposes the same way - it excludes optional headers. Since 
the base TCP header is 20 bytes and the IPv4 header is 20 bytes, the 
standard MSS for TCP over IPv4 is PMTU - 40.   SCTP would have a 
standard MSS 8 bytes higher (PMTU - 32 for SCTP over IPv4).
Of course setting IP options reduces the effective MSS in both cases.

- Mark

---

From: Randall Stewart <[email-masked]>
Date: Fri, 30 Dec 2005 08:01:07 -0500

> knowledge of the true PMTU, it can reduce the message size by itself.

You miss my point I think..

A RDDP implementation that DOES NOT want fragmentation will
do the following:

setsockopt(...,SCTP_DISABLE_FRAGMENTS, on=1, sizeof(int));
setsockopt(...,SCTP_MAXSEG, sz=sizeof_there_segment, sizeof(int));

Now, what will happen is they will get no more tha size_there_segment
chunks and all in a single message.. no fragmentation will occur.

> only the fragment size. 

But you combine it with no fragmentation and you get only
single sized msgs of your prescribed size (at least
limited to that)..

Now I agree with Shridhar.. having a NO_BUNDLE option might
also help here... but I am still waiting to here what
Kacheong has to say as well ;-D

The primary motivation for SCTP is reduced
> purposes is to set SCTP_NODELAY.

That will NOT disable bundling.. consider

cwnd=3000 (intially)

while(true)
send (100)--->

Now it is true the first 30 or so messages will
hit the wire.. but after that you will queue
up messages to be sent.. and they WILL be bundled..

R

---

From: Mark Butler <[email-masked]>
Date: Fri, 30 Dec 2005 16:15:25 -0700

Randall Stewart wrote:

> Mark Butler wrote:
>
>> The RDDP SCTP draft says that fragmentation at the SCTP layer should 
>> be avoided.  However the SCTP_MAXSEG option cannot be used to get 
>> _less_ SCTP layer fragmentation, only _more_ SCTP layer 
>> fragmented.  This can be done with a getsockopt(...,SCTP_STATUS,...) 
>> call.  If the RDDP layer has better knowledge of the true PMTU, it 
>> can reduce the message size by itself.
>
>

I do not understand why one would want to set SCTP_MAXSEG if 
fragmentation is disabled, unless as a crutch to make sure that you 
cannot send messages larger than the specified size.

If a user actually wants to send data with fragmentation disabled, he 
should read the current fragmentation point and reduce all sends to be 
less than or equal to that size.   Otherwise he/she  is liable to get 
immediate rejections and have to guess what the maximum is. And of 
course if SCTP fragmentation is disabled, the stack may have to fragment 
at the IP layer if the known PMTU goes down (another reason for MSS 
negotiation).  That limits the utility of SCTP_DISABLE_FRAGMENTS - 
generally speaking it is certainly better to fragment at the SCTP layer 
if  necessary than at the IP layer.

By the way, the SCTP_DISABLE_FRAGMENTS documentation says an immediaate 
error will be returned if a message if exceeds the current PMTU size.  
This should say the current fragmentation point I believe.  The MTU is 
defined at the IP layer right?  Or is it the SCTP layer?

> Kacheong has to say as well ;-D

Is an implementation really supposed to send ECNEs in separate 
messages?  Normally ECNE only rides with packets containing other 
chunks.  So I agree such an option is well defined, but why would anyone 
want it? 

>> fragmentation.  You can certainly still fragment if you do not 
>> bundle.  If one does not want bundling, the best way to disable it 
>> for practical purposes is to set SCTP_NODELAY.
>
>
> That will NOT disable bundling.

I stand corrected (I meant most of the time, that is not the case either).

- Mark