Subject: Apparent SCTP API oversights, draft-ietf-tsvwg-sctpsocket-30

From: "Dan Wing" <[email-masked]>
Date: Thu, 4 Aug 2011 17:05:22 -0700

I would like clarification on my misunderstanding, or changes
to draft-ietf-tsvwg-sctpsocket, to accommodate what appear to
be some oversights in the SCTP API:

1. The infotype parameter of sctp_sendv() allows indicating if the 
data should be sent with partial reliability or full reliability
using sctp_prinfo.  However, there does not appear to be a way to 
adjust the IPv4 or IPv6 DSCP bits when switching between 
full/partial reliablity, except by modifying the entire
association itself (using spp_ipv4_tos).  This is quite limiting.

2. When modifying an association (e.g., to have different DSCP
value), I could not find text in draft-ietf-tsvwg-sctpsocket-30 that
indicates if the modification applies immediately (including to 
packets queued and to packets not yet acknowledged by the 
receiver) or if the modification applies to subsequent calls
to send().

This means that to attempt to work around (1), it isn't clear
if the API would permit me to modify the association's DSCP
value, call send(), and know that payload will be sent with
the desired DSCP value.

3. Related to (1), it is desirable that full reliable packets not
be bundled with partial reliability packets, so that they can have
different DSCP markings.  

http://tools.ietf.org/html/rfc4960#section-10.1 mentions a no-bundle
flag.  However, I could not find in draft-ietf-tsvwg-sctpsocket-30 
where this is controlled; it appears the API lacks this ability.

4. draft-ietf-tsvwg-sctpsocket-30 uses the archaic term ToS
rather than DSCP.  At a minimum, to ease string searching, please 
add text near spp_ipv4_tos something like "This is also called the
DSCP field [RFC2474]".  I would prefer, though, that 
draft-ietf-tsvwg-sctpsocket-30 use the term DSCP in compliance 
with RFC3260 and other DiffServ documents.

5. I believe it is an oversight that draft-ietf-tsvwg-sctpsocket-30 
provides a way to adjust the IPv4 ToS (spp_ipv4_tos) and the IPv6
flow label (spp_ipv6_flowlabel), but lacks a way to query or adjust
the IPv6 Traffic Class field.  Please correct this oversight or
add text justifying the inability to adjust the IPv6 Traffic Class
field.

-d

---

From: Michael Tuexen <[email-masked]>
Date: Fri, 5 Aug 2011 22:58:34 +0200

Hi Dan,

thank you very much for your comments.
See my answers in-line.

Best regards
Michael

> association itself (using spp_ipv4_tos).  This is quite limiting.
A couple of points:
* The DSCP stuff applies to packets, not to user messages. Any
  send() call applies to a user message. So this does not really
  fit.
* The sendv() is a convenience function, so you can't do everything
  with it. The generic way is by using CMSGs.
* The DCSP stuff is related to the IP layer, not the SCTP layer.
* I'm also not sure if you can easily map PR-SCTP policies to
  DCSP code points. You might want to do that for particular
  usages of particular policies, but in general a mapping seems
  to be hard.
* For IPv6 you can do that by using the IPV6_TCLASS CMSG.
* For IPv4 there is no CMSG for specifying the TOS (as far
  as I know). But adding such a CMSG at the IPv4 layer would
  be the way to address this.
> value, call send(), and know that payload will be sent with
At least for the FreeBSD implementation this applies to all
packets (transmitted or retransmitted) sent after the socket option
has been set. The values do NOT stick to user messages.
> be bundled with partial reliability packets, so that they can have
... which means the workaround doesn't really work...
> where this is controlled; it appears the API lacks this ability.
The functionality is provide by the SCTP_NODELAY socket option.
> with RFC3260 and other DiffServ documents.
I'm happy to change this text. However, the document has passed
the WG LC. Not sure how to handle this? WG chairs?
We can either reissue he document with the small changes discussed
in this mail or handle it as comments during the IESG review.
Up to the chairs.
> add text justifying the inability to adjust the IPv6 Traffic Class
Hmmm. You have a point here...
It is hard to change he structure anymore since implementations are
out... However, the spp_ipv6_flowlabel is 32-bit, enough space to
contain the flowlabel and the DSCP. We could add text how to [sg]et
the DSCP.
Kacheong: Do you have some suggestions here?

---

From: "Dan Wing" <[email-masked]>
Date: Fri, 5 Aug 2011 16:36:26 -0700

> -----Original Message-----
> From: Michael Tuexen [mailto:[email-masked]]
> Sent: Friday, August 05, 2011 1:59 PM
> To: Dan Wing; [email-masked]; James M. Polk
> Cc: [email-masked]; [email-masked];
> [email-masked]; Peter Lei (peterlei); [email-masked]
> Subject: Re: Apparent SCTP API oversights, draft-ietf-tsvwg-sctpsocket-
> 30
> 
> Hi Dan,
> 

To me this seems necessary, or else SCTP's ability to carry
multiple streams (of different application delivery precedence, e.g.,
partial reliability) cannot influence the network's drop precedence.
Without this capability the only workable alternative, if someone 
wants the  network to know the drop precedence for a packet, is 
to create two separate SCTP associations and carry the reliable stream 
in one association and the unreliable stream in the other association.
Creating a separate connection with certain DSCP values is what we 
do with UDP and TCP today.  This limitation seems the antithesis of 
SCTP's design which allows a single SCTP association to carry several 
streams where each stream can have different retransmit characteristics.

> has been set. The values do NOT stick to user messages.

This seems significant, and means an application cannot work
around the missing functionality in the API.  Can this be 
added to the document?

> The functionality is provide by the SCTP_NODELAY socket option.

Can the word "bundle" be added somewhere in the description of
SCTP_NODELAY?  Such consistency of terminology eases searching for
things like this.

> contain the flowlabel and the DSCP. We could add text how to [sg]et

That seems a reasonable solution which makes only a small change
to existing code.

-d

---

From: Michael Tuexen <[email-masked]>
Date: Sun, 7 Aug 2011 16:50:23 +0200

> partial reliability) cannot influence the network's drop precedence.
If you want a per user message DSCP, that is required. However,
we don't define any IPv[46] layer socket option or CMSG in the
SCTP socket API ID.
> streams where each stream can have different retransmit characteristics.
Hmm. Assuming you would send user messages with a limited lifetime of,
let's say, 5 seconds. What DSCP would you use?

For different PR-SCTP policies, this might be clearer. For example,
FreeBSD allows to limit the number of retransmissions. For limiting
it to 0, it might be clear, but what would be an appropriate DSCP
with you allow, let's say, 3 retransmissions?

Or do you have a different PR-SCTP policy in mind?

In general I'm wondering how SCTP behaves if you use different
DSCP within the same SCTP association. Not that it might break
the association, but I'm not sure about interactions with the
CC.
> around the missing functionality in the API.  Can this be 
Up to the chairs...
> That seems a reasonable solution which makes only a small change
However, this does not solve your problem of being able to provide a
per packet DSCP. You can do this with the IPV6_TCLASS cmsg for
IPv6, but I'm not aware of something similar for IPv4. 
Maybe the reason is that for UDP you can do this my using a setsockopt()
call before each send() call. I'm not sure if it makes sense for TCP
to use multiple DSCPs per TCP connection.

Best regards
Michael

---

From: Randy Stewart <[email-masked]>
Date: Mon, 8 Aug 2011 07:27:22 -0400

Comments in-line...

> the association, but I'm not sure about interactions with the

There was a group when I worked at Cisco that wanted to do this, I don't
remember the reasons, but I pointed out that:

1) Each flow packet using  a  different DSCP or flow-label will appear
   in the network as having a different path (who knows how much
   different they will be .. but they will be different)

2) Since they travel different "paths" in the network, they will have
   varying RTT times.

3) They will also arrive in completely different order.

The result of this is:

a) You will get false fast retransmits
b) You MAY get unexpected HOL
c) You will generate a lot more SACK's (since holes would
   be common which normally do not do delayed sack in 
   the case of a hole).

So bottom line is its probably a bad idea.. better to setup separate associations
and just go ahead and route the traffic on the different associations for
the different priorities.

I guess if you can live with the drawbacks, it would be ok. You could
always do the sort of CMT like things that are done when CMT is on and
keep a psuedo cumulative ack on each of your separate paths.. this would
be a *LOT* of work.. but it could be done

> to use multiple DSCPs per TCP connection.

Which would have the same effects for it that I list above
for SCTP.. only worse, since HOL would be set in all the
time... SCTP would escape some of it depending on what streams
were used and if unordered is thrown in the mix here too..

Also, thinking on this, how SCTP sends its FWD-TSN (what codepoint/path is
used) would become an interesting mixer to the items I listed above.

R

>>> Kacheong: Do you have some suggestions here?

-----
Randall Stewart
[email-masked]

---

From: Qiaobing Xie <[email-masked]>
Date: Mon, 08 Aug 2011 11:49:32 -0500

Dan,

A different way to look at this is - the overall data delivery 
reliability between two SCTP endpoints has two control points; 1) 
selectively dropping packets along the path (ie, per-hop control), and 
2) selectively retransmitting lost packets by the sending endpoint (ie, 
end-to-end control). As a transport layer protocol, SCTP naturally is 
designed to only do the latter. To ask SCTP to manage both would be 
architecturally awkward IMO.

-Q

---

From: "Dan Wing" <[email-masked]>
Date: Mon, 8 Aug 2011 11:21:48 -0700

> Sent: Sunday, August 07, 2011 7:50 AM
> To: Dan Wing
> Cc: [email-masked]; 'James M. Polk'; draft-ietf-tsvwg-
> [email-masked]; [email-masked]; [email-masked]; 'Peter
> Lei (peterlei)'; [email-masked]

The value depends on that network's configuration.  RFC4594 provides
some guidance on different values.

> > around the missing functionality in the API.  Can this be

?  As in, this vagueness in the API is acceptable to you?

> per packet DSCP.

Agreed.  But at least it brings IPv6 on par with IPv4.

> You can do this with the IPV6_TCLASS cmsg for

I agree it makes little sense for TCP to change DSCP values of
a flow.  But SCTP isn't TCP, of course.

-d

---

From: Michael Tuexen <[email-masked]>
Date: Tue, 9 Aug 2011 11:32:06 +0200

> some guidance on different values.
Hmmm. I could not find enough guidance to choose the right value if
the timeout is 5 seconds compared to 1 second compared to 10 seconds.

But as I said and Randy wrote in more detail. I'm not sure if
it is wise to use multiple DSCP per path. So fixing the sctp_paddrparams
is helpful and within the scope of the document. I'll prepare a proposed
change of text. If you want per user message behavior you need
an IP level cmsg which is not within the scope of this document.
> ?  As in, this vagueness in the API is acceptable to you?
I'll send out a proposed text change later. But the document is *after* WG LC.
So we have two choices:
1. After agreeing that the suggested text resolves your issues, resubmit
   the ID. Progress the new version to the IESG.
2. Handle your comments as comments received within the IESG review.
I have done both (with different documents) depending on what the WG chairs
prefer.
> Agreed.  But at least it brings IPv6 on par with IPv4.
Yes.

---

From: Michael Tuexen <[email-masked]>
Date: Tue, 9 Aug 2011 23:07:09 +0200

Hi Dan,

based on the discussion, I suggest the following text changes:

> association itself (using spp_ipv4_tos).  This is quite limiting.
No changes.
> value, call send(), and know that payload will be sent with
I added at the end of section 8.1.12:
<t>Please note that changing the flowlabel or DSCP value will affect
all packets sent by the SCTP stack after setting these parameters.
The flowlabel might also be set via the sin6_flowinfo field of the
struct socketaddr_in6 structure.</t>
> where this is controlled; it appears the API lacks this ability.
I added to section 8.1.5:
In particular, not using any Nagle-like algorithm might
reduce the bundling of small user messages in cases where this
would require an additional delay.</t>
> add text justifying the inability to adjust the IPv6 Traffic Class
The stuff now reads:
struct sctp_paddrparams {
  sctp_assoc_t spp_assoc_id;
  struct sockaddr_storage spp_address;
  uint32_t spp_hbinterval;
  uint16_t spp_pathmaxrxt;
  uint32_t spp_pathmtu;
  uint32_t spp_flags;
  uint32_t spp_ipv6_flowlabel;
  uint8_t spp_dscp;
};

<t hangText="spp_ipv6_flowlabel:">
This field is used in conjunction with the SPP_IPV6_FLOWLABEL flag
and contains the IPv6 flowlabel. The 20 least significant bits are
used for the flowlabel.
This setting has precedence over any IPv6 layer setting.</t>
<t hangText="spp_dscp:">
This field is used in conjunction with the SPP_DSCP flag and
contains the Differentiated Services Code Point (DSCP).
The 6 least significant bits are used for the DSCP.
This setting has precedence over any IPv4 or IPv6 layer setting.</t>

...
<t hangText="SPP_IPV6_FLOWLABEL:">
Setting this flag enables the setting of the IPV6 flowlabel value.
The value is contained in the spp_ipv6_flowlabel field.
<vspace blankLines="1"/>
Upon retrieval, this flag will be set to indicate that the spp_ipv6_flowlabel
field has a valid value returned. If a specific destination address is
set (in the spp_address field), then the value returned is that
of the address. If just an association is specified (and no address), then
the association's default flowlabel is returned. If neither an association nor a
destination is specified, then the socket's default flowlabel is returned.
For non IPv6 sockets, this flag will be left cleared.</t>
<t hangText="SPP_DSCP:"> Setting this flag enables the setting of
the DSCP value associated with
either the association or a specific address.
The value is obtained in the spp_dscp field.
<vspace blankLines="1"/>
Upon retrieval, this flag will be set to indicate that the spp_dscp field
has a valid value returned. If a specific destination address is set when
called (in the spp_address field) then that specific destination address'
DSCP value is returned. If just an association is specified then
the association default DSCP is returned. If neither an association nor a
destination is specified, then the sockets default DSCP is returned.</t>

Does this address your issues?

Best regards
Michael

---

From: "Dan Wing" <[email-masked]>
Date: Tue, 9 Aug 2011 15:35:30 -0700

Yes, thanks.  

With the datastructure renamed from spp_ipv4_tos to spp_dscp, I assume you
would also change prose from ToS to DSCP?

#grep -nE "tos|ToS|TOS" draft-ietf-tsvwg-sctpsocket-30.txt
3825:     uint8_t spp_ipv4_tos;
3872:   spp_ipv4_tos:  This field is used in conjunction with the
3873:      SPP_IPV4_TOS flag.  This setting has precedence over any IPv4
3924:      SPP_IPV4_TOS:  Setting this flag enables the setting of the IPV4
3925:         TOS value associated with either the association or a specific
3926:         address.  The value is obtained in the spp_ipv4_tos field.
3929:         spp_ipv4_tos field has a valid value returned.  If a specific
3931:         field) then that specific destination address' TOS value is
3933:         association default TOS is returned.  If neither an
association
3934:         nor a destination is specified, then the sockets default TOS
is

-d

> Sent: Tuesday, August 09, 2011 2:07 PM
> To: Dan Wing
> [email-masked]; [email-masked]; 'Peter Lei
> (peterlei)'; [email-masked]

---

From: Michael Tuexen <[email-masked]>
Date: Wed, 10 Aug 2011 06:57:51 +0200

> 3934:         nor a destination is specified, then the sockets default TOS
Yes, that is the text at the end of my e-mail...

Best regards
Michael