Subject: [tsvwg] SCTP NDATA draft

From: "Mo Zanaty (mzanaty)" <[email-masked]>
Date: Wed, 29 Oct 2014 01:18:53 +0000

Hi,
While reviewing this draft for becoming mandatory in rtcweb data channels, I see two issues, which are independent of rtcweb.

1. If negotiated, NDATA must be used for all messages. The NDATA format requires 8 more bytes than DATA (for MID+FSN). But these extra 8 bytes are only useful for fragmented messages. To avoid this overhead for small non-fragmented messages, can we relax / optimize this rule to be NDATA must be used for all *fragmented* messages, while non-fragmented messages continue to use DATA?

Proposed change to the next-to-last paragraph of section 1.1:

OLD:
   The support of the N-DATA chunk is negotiated during the association
   setup using the Supported Extensions Parameter as defined in
   [RFC5061].  If N-DATA support has been negotiated for an association
   N-DATA chunks are used for all user-messages and no DATA chunks.  It
   should be noted, that an SCTP implementation needs to support the
   coexistence of associations using DATA chunks and associations using
   N-DATA chunks.

NEW:
   The support of the N-DATA chunk is negotiated during the association
   setup using the Supported Extensions Parameter as defined in
   [RFC5061].  If N-DATA support has been negotiated for an association
   N-DATA chunks are used for all fragmented user messages, and
   DATA chunks are used for all non-fragmented user messages.  It
   should be noted, that an SCTP implementation needs to support the
   coexistence of associations using DATA chunks and associations using
   N-DATA chunks.

2. Fragmented messages can’t be interleaved. This significantly dilutes the value of NDATA. Apps that truly care about head of line blocking can’t rely on NDATA at all. They must implement app layer fragmentation with messages well below common MTUs to avoid blocking, hence defeating NDATA. NDATA is arguably harmful for these apps, since it is 8 bytes closer to exceeding the MTU, forcing SCTP fragments and hence risking HOL blocking. To avoid this, can we relax this rule to allow interleaving fragmented messages, since MID is already part of NDATA?

Proposed change to the last paragraph of section 3.1:

OLD:
   The sender MUST NOT have more than one ordered fragmented message
   being produced in any one stream.  The sender MUST NOT have more than
   one un-ordered fragmented message being produced in any one stream.
   The sender MAY have one ordered and one unordered fragmented message
   being produced within a single stream.  At any time multiple streams
   MAY be producing an ordered or unordered fragmented message.

NEW:
   The sender MAY have more than one ordered fragmented message
   being produced in any one stream.  The sender MAY have more than
   one un-ordered fragmented message being produced in any one stream.
   The sender MAY have multiple ordered and unordered fragmented messages
   being produced within a single stream.  At any time multiple streams
   MAY be producing multiple ordered or unordered fragmented messages.

Regards,
Mo

---

From: Karen Elisabeth Egede Nielsen <[email-masked]>
Date: Wed, 29 Oct 2014 08:50:09 +0100

Hi Mo,

Please find feedback from my side below.

BR, Karen

On 29 October 2014 02:18, Mo Zanaty (mzanaty) <[email-masked]> wrote:

>  Hi,
> While reviewing this draft for becoming mandatory in rtcweb data channels,
> I see two issues, which are independent of rtcweb.
>
>  1. If negotiated, NDATA must be used for all messages. The NDATA format
> requires 8 more bytes than DATA (for MID+FSN). But these extra 8 bytes are
> only useful for fragmented messages. To avoid this overhead for small
> non-fragmented messages, can we relax / optimize this rule to be NDATA must
> be used for all *fragmented* messages, while non-fragmented messages
> continue to use DATA?
>
> This will cost in performance in SCTP SW  - optionality costs in the
processing paths.
But it is true that the extra 8 bytes also costs some. Are you mostly
concerned about the 8 bytes from a running cache footprint perspective or
from
a wire perspective ?

> *   N-DATA chunks are used for all user-messages and no DATA chunks.*  It
> *   N-DATA chunks are used for all fragmented user messages, and*
> *   DATA chunks are used for all non-fragmented user messages.  *It
>  2. Fragmented messages can’t be interleaved.
>

This is not correct. They will be interleaved with all messages sent on
other streams.
Usage of multiple streams is really a prerequisite to avoid HOL. Further,
usage of multiple streams is
the best way to enforce differences in the priority of different messages.

Personally I would have like for there to be closer coupling in between
ndata and SCTP-PR in that I would like for
the message priority (of SCTP-PR) to be inherited from the stream priority
if nothing else is specified.
I have made this request towards the ndata author, but it may have been off
list.

This significantly dilutes the value of NDATA.

Do you require to use one stream only ?

Apps that truly care about head of line blocking can’t rely on NDATA at all.

Se above.

They must implement app layer fragmentation with messages well below common
> MTUs to avoid blocking, hence defeating NDATA. NDATA is arguably harmful
> for these apps, since it is 8 bytes closer to exceeding the MTU, forcing
> SCTP fragments and hence risking HOL blocking. To avoid this, can we relax
> this rule to allow interleaving fragmented messages, since MID is already
> part of NDATA?
>
We already have the stream concept why do we need now also a new concept.
Also we would then need to control which and what to control the
interleaving with.I.e. what to give priority. Yes perhaps we could use the
message priority, but the  right thing in SCTP is to use the stream concept
which is already there !!

>     The sender *MUST NOT* have more than one ordered fragmented message
>    being produced in any one stream.  The sender *MUST NOT* have more than
>    The sender MAY have *one ordered and one unordered fragmented message*
>    MAY be producing *an ordered or unordered fragmented message*.
>
>  NEW:
>     The sender *MAY* have more than one ordered fragmented message
>    being produced in any one stream.  The sender *MAY* have more than
>    The sender MAY have *multiple ordered and unordered fragmented
> messages*
>    MAY be producing *multiple ordered or unordered fragmented messages*.
>
>  Regards,
> Mo
>
>

---

From: "Mo Zanaty (mzanaty)" <[email-masked]>
Date: Wed, 29 Oct 2014 19:33:05 +0000

Hi Karen,

I understand the design intent was to avoid one stream blocking another. However, I think that misses the true application need. It assumes applications only care about blocking of non-fragmented messages and don’t care about blocking of fragmented messages.

I think that assumption is wrong for most applications. It is certainly wrong for rtcweb applications, where JSON objects would be a natural message payload for JS apps, and these can easily exceed common MTUs depending on the complexity and verbosity of the JSON object.

If an application does not care about blocking, it does not need NDATA. If it cares, it must create a new stream for every potentially fragmented message (which requires guessing the MTU) since that will block subsequent fragmented messages. It must also have a garbage collector to close those streams (or otherwise mark them reusable) after ensuring delivery with an app layer ack.

That is far more overhead and complexity for both the application and the SCTP stack than simply implementing application layer fragmentation which defeats NDATA. Even more true for rtcweb applications since they have the extra overhead of DCEP to open/close bidirectional channels.

So applications which truly care about blocking will almost certainly implement application layer fragmentation rather than relying on NDATA (which will block) or jumping through hoops to create new streams for each potentially fragmented message.

It should be simple to fix NDATA to meet the true application need. The text change is dead simple. There is no protocol change, since MID is already part of NDATA. There is only implementation effort, which should be limited to maintaining FSN state for multiple MIDs per stream rather than only one. That seems like a small effort for making NDATA actually useful for most applications.

Cheers,
Mo

On 10/29/14, 3:50 AM, Karen Elisabeth Egede Nielsen <[email-masked]<mailto:[email-masked]>> wrote:

Hi Mo,

BR, Karen

On 29 October 2014 02:18, Mo Zanaty (mzanaty) <[email-masked]<mailto:[email-masked]>> wrote:
Hi,
While reviewing this draft for becoming mandatory in rtcweb data channels, I see two issues, which are independent of rtcweb.

This will cost in performance in SCTP SW  - optionality costs in the processing paths.
But it is true that the extra 8 bytes also costs some. Are you mostly concerned about the 8 bytes from a running cache footprint perspective or from
a wire perspective ?

OLD:

NEW:

This is not correct. They will be interleaved with all messages sent on other streams.
Usage of multiple streams is really a prerequisite to avoid HOL. Further, usage of multiple streams is
the best way to enforce differences in the priority of different messages.

Personally I would have like for there to be closer coupling in between ndata and SCTP-PR in that I would like for
the message priority (of SCTP-PR) to be inherited from the stream priority if nothing else is specified.
I have made this request towards the ndata author, but it may have been off list.

Se above.

They must implement app layer fragmentation with messages well below common MTUs to avoid blocking, hence defeating NDATA. NDATA is arguably harmful for these apps, since it is 8 bytes closer to exceeding the MTU, forcing SCTP fragments and hence risking HOL blocking. To avoid this, can we relax this rule to allow interleaving fragmented messages, since MID is already part of NDATA?
We already have the stream concept why do we need now also a new concept. Also we would then need to control which and what to control the interleaving with.I.e. what to give priority. Yes perhaps we could use the message priority, but the  right thing in SCTP is to use the stream concept which is already there !!

OLD:

NEW:

Regards,
Mo

---

From: Karen Elisabeth Egede Nielsen <[email-masked]>
Date: Wed, 29 Oct 2014 22:01:21 +0100

Hi Mo,

I am all for finding a solution that fits the needs of the application.
Just so that this is clear.

So you are actually not looking for ndata to support the interleaving
property (or that perhaps also, never mind)
but mainly to prevent RX and TX blocking for unordered delivery of larger
than PMTU messages,
possibly also for ordered delivery of such messages.

OK. If others agree then I suppose that ndata can be extended as you
suggest.
But reassembly costs, of course. Especially over a best effort network.
SCTP will have to filter and manage concurrent reassembly on yet one more
identifier/dimension.

The application need not guess the MTU, the SCTP fragmentation point can be
read on the socket (yes it may vary, but it will not jump that much around
I suppose). In special cases where the application is anyway chopping up
the messages it will be wildly best to do this in subPMTU sizes.

BR, karen

On 29 October 2014 20:33, Mo Zanaty (mzanaty) <[email-masked]> wrote:

>  Hi Karen,
>
>  I understand the design intent was to avoid one stream blocking another.
> However, I think that misses the true application need. It assumes
> applications only care about blocking of non-fragmented messages and don’t
> care about blocking of fragmented messages.
>
>  I think that assumption is wrong for most applications. It is certainly
> wrong for rtcweb applications, where JSON objects would be a natural
> message payload for JS apps, and these can easily exceed common MTUs
> depending on the complexity and verbosity of the JSON object.
>
>  If an application does not care about blocking, it does not need NDATA.
> If it cares, it must create a new stream for every potentially fragmented
> message (which requires guessing the MTU) since that will block subsequent
> fragmented messages. It must also have a garbage collector to close those
> streams (or otherwise mark them reusable) after ensuring delivery with an
> app layer ack.
>
>  That is far more overhead and complexity for both the application and
> the SCTP stack than simply implementing application layer fragmentation
> which defeats NDATA. Even more true for rtcweb applications since they have
> the extra overhead of DCEP to open/close bidirectional channels.
>
>  So applications which truly care about blocking will almost certainly
> implement application layer fragmentation rather than relying on NDATA
> (which will block) or jumping through hoops to create new streams for each
> potentially fragmented message.
>
>  It should be simple to fix NDATA to meet the true application need. The
> text change is dead simple. There is no protocol change, since MID is
> already part of NDATA. There is only implementation effort, which should be
> limited to maintaining FSN state for multiple MIDs per stream rather than
> only one. That seems like a small effort for making NDATA actually useful
> for most applications.
>
>  Cheers,
> Mo
>
>
>   On 10/29/14, 3:50 AM, Karen Elisabeth Egede Nielsen <
> [email-masked]> wrote:
>
>   Hi Mo,
>
>> While reviewing this draft for becoming mandatory in rtcweb data
>> channels, I see two issues, which are independent of rtcweb.
>>
>  They must implement app layer fragmentation with messages well below
>> common MTUs to avoid blocking, hence defeating NDATA. NDATA is arguably
>> harmful for these apps, since it is 8 bytes closer to exceeding the MTU,
>> forcing SCTP fragments and hence risking HOL blocking. To avoid this, can
>> we relax this rule to allow interleaving fragmented messages, since MID is
>> already part of NDATA?
>>

---

From: "Mo Zanaty (mzanaty)" <[email-masked]>
Date: Thu, 30 Oct 2014 01:36:36 +0000

Hi authors,

I assume MID is a serial message id incremented for each message (ordered or unordered) on the stream, and was added because SID+SSN is insufficient as a serial message id for unordered messages. Is that assumption correct? If so, it would be good text to add, since the draft does not define MID or describe its use. If incorrect, what is MID for?

Regards,
Mo

On 10/29/14, 5:01 PM, Karen Elisabeth Egede Nielsen <[email-masked]<mailto:[email-masked]>> wrote:

Hi Mo,

I am all for finding a solution that fits the needs of the application. Just so that this is clear.

So you are actually not looking for ndata to support the interleaving property (or that perhaps also, never mind)
but mainly to prevent RX and TX blocking for unordered delivery of larger than PMTU messages,
possibly also for ordered delivery of such messages.

OK. If others agree then I suppose that ndata can be extended as you suggest.
But reassembly costs, of course. Especially over a best effort network.
SCTP will have to filter and manage concurrent reassembly on yet one more identifier/dimension.

The application need not guess the MTU, the SCTP fragmentation point can be read on the socket (yes it may vary, but it will not jump that much around I suppose). In special cases where the application is anyway chopping up the messages it will be wildly best to do this in subPMTU sizes.

BR, karen

On 29 October 2014 20:33, Mo Zanaty (mzanaty) <[email-masked]<mailto:[email-masked]>> wrote:
Hi Karen,

Cheers,
Mo

Hi Mo,

BR, Karen

OLD:

NEW:

Se above.

OLD:

NEW:

Regards,
Mo

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Wed, 29 Oct 2014 19:50:13 -0600

Mo,

Mo Zanaty (mzanaty) wrote:                 (Thu, 30 Oct 2014 01:36:36)
>    Hi authors,
> 
>    I assume MID is a serial message id incremented for each message
>    (ordered or unordered) on the stream, and was added because SID+SSN is
>    insufficient as a serial message id for unordered messages. Is that
>    assumption correct? If so, it would be good text to add, since the
>    draft does not define MID or describe its use. If incorrect, what is
>    MID for?

I think that it was more becuase the B and E bits are insufficient to
identify an interior fragment (ordererd or unordered).

Nevertheless, I agree that fragments from one stream should not block
fragments from another (ordered or otherwise).  I even think that an
unordered fragment should not block another message in the same stream.

That is a +1 for your changes.

There was no need for that restriction in the first place for senders on
ordered streams: NDATA (or whatever we are calling it now) simply makes
it easier for the receiver to do partial delivery and avoids some degree
of blocking at the receiver.  I agree that it should also remove all
restrictions from unordered message both within and between streams.  But
we should be careful that the language is still consistent throughout the
document.

--brian

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

From: "Mo Zanaty (mzanaty)" <[email-masked]>
Date: Thu, 30 Oct 2014 02:03:27 +0000

I thought FSN was for serializing interior fragments while MID was for
serializing all messages on the stream. Or maybe even globally across all
streams like TSN? All I know for sure is it is a Message Identifier!  ;)

Mo

On 10/29/14, 9:50 PM, Brian F. G. Bidulock <[email-masked]> wrote:

Mo,

--brian

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Wed, 29 Oct 2014 22:59:03 -0600

Mo,

Mo Zanaty (mzanaty) wrote:         (Thu, 30 Oct 2014 02:03:27)
> streams like TSN? All I know for sure is it is a Message Identifier!  ;)

Sorry, I reversed them...  Nevertheless, I still agree with your
objective of not blocking any fragments with any other fragments
as there is sufficient information to avoid blocking in all cases.

--brian

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/