Subject: [rddp] RE: I-D ACTION:draft-stewart-rddp-sctp-01.txt

From: "Williams, Jim" <[email-masked]>
Date: Mon, 16 Dec 2002 13:56:19 -0500

Some quick thoughts:

>   The DDP layer is responsible for submitting messages to the SCTP
>   layer that conform to the PMTU.  The SCTP layer MAY assume this has
>   been done.  The DDP layer MAY also wish to set the fragmentation
>   point (usually termed SCTP_MAXSEG) to a low enough value to assure
>   that PMTU changes will not cause IP fragmentation.

I don't have a problem with the "MAY" here, but usually that
won't be a good idea.  Receiver performance will usually be optimized
by maximizing the fragment size.  Keeping all the chunk headers
up front for locality of reference can only help, too.

IP fragmentation on dynamic PMTU reductions (infrequent) seems
like a small price to pay.  Probing for larger PMTUs seems like
a good application for multiple SCTP chunks.  Is there a NOP
type chunk that could be used to pad out the PMTU to probe
for larger size without needing to bother the DDP layer
until the larger PMTU is actually observed (usually it won't
be)?

>   This mapping places an entire SCTP association into a specific DDP
>   mode: DDP or DDP+RDMA.  It is presumed that the handling of incoming
>   data chunks for DDP enabled associations is sufficiently different
>   than for routine SCTP associations that it is undesirable to mix DDP
>   and non-DDP streams in a single association.  An application that
>   needs to mix DDP and non-DDP traffic must use more than a single
>   association.

This is not completely clear to me.  Does this mean that if the DDP+RDMA
adaption indication is done for a specific association, then ALL traffic
over that association is DDP+RDMA, or does it mean that DDP+RDMA may be
negotiated and used by applications, with initial negotiation done in 
non-DDP mode?  The former seems much clearer and less ambiguous, but
not sure if that is how SCTP defines adoption indications.

Using the 32 bit Payload Protocol Identifier to carry one bit of
information looks less than optimal.  Also, with the current definition,
the DDP and RDMA headers end up being unaligned.  I think some additional
thought should go into optimizing the combined header format.

Regarding ordering, I would strongly support the preferred option
of using unordered delivery.  The argument can (and has) been made
that it is fully compliant with ordered transport semantics to
do out of order placement, however there will be counter arguments.
Some of those counter arguments actually seem to me to have 
some merit.  Regardless, even if those arguments could be won,
I don't see the down side of using unordered delivery.  If nothing
else it should save a lot of time and reduce risk of rejection.

I would suggest that using SCTP ordered delivery be considered
as a mechanism for providing IN ORDER placement and
IN ORDER delivery.  This could be an optional mode, but there
may be applications where this is of significant value.

Regarding the choice between option 5.1 and 5.2, I would 
slightly favor 5.2 for its simplicity.  However, only if
this can be done without any interoperability problems with
normal SCTP implementations.

Section 7.2 looks incomplete to me.  Error shutdown and
resetting of a stream needs to end to end coordination and
requires a state machine on each end.  Maybe it's simpler
than I think, but I think it should be documented more
clearly.  Overall, I have some concern whether the
complexity is justified by the value.

Section 7.1 table entry "DDP+RDMA" looks slightly off.
> If the local endpoint  only indicated
>                  RDMA, then ONLY the procedures in
>                  [RDMA-Draft] MUST be followed.

Should be DDP rather than RDMA.  Formatting needs to be
tweaked also.
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: Caitlin Bestler <[email-masked]>
Date: Mon, 16 Dec 2002 15:54:22 -0600

Thanks for the quick feedback Jim, many excellent points.

> not sure if that is how SCTP defines adoption indications.

The intent is to facilitate implementing DDP/SCTP in the RNIC, while passing any
non-DDP SCTP to the host for processing.

If the entire association is in a single mode, then the only required
co-ordination is to know which associations are in which modes. The RNIC must
pass associations it does not control to the default SCTP stack. The default
SCTP stack must refrain from generating traffic for the RNIC's associations.

The option to restrict an entire endpoint, not just an association, takes that
simplification even farther. It provides some slight benefits on accepting
connections. Very sligh, in my estimation, which is why it is only a MAY.
At the minimum, it would simplify validation testing.

In any event, SCTP provides for 64K endpoints off of each IP address. An
association is a pairing of a local endpoint with a remote endpoint. So it is
not as though requiring a second association for non-DDP traffic would be a
problem.

Setting DDP or DDP+RDMA mode on a per stream basis might be nice, but would
require heavy co-ordination on ACKing and TSN tracking between the DDP/SCTP in
the RNIC and the default SCTP stack.

Given that complexity, and the low probability of the same pair of processes
exchaning both DDP/SCTP and non-DDP/SCTP traffic, the restriction to place an
entire association in one mode makes sense to me.

> information looks less than optimal.  Also, with the current
> definition, the DDP and RDMA headers end up being unaligned.  I think
> some additional thought should go into optimizing the combined header
> format.
> 
Good suggestion. We could define a filler before the DDP command.

Using the 32 bit Payload Protocol Identifier to carry one bit is indeed less
than optimal, but the alternative use of the Payload Protocol Identifier would
be to carry zero bits of information.

The Payload Protocol Identifier would be a good place to move the DDP Queue ID,
but doing so would require changes to both DDP and MPA. It didn't seem
worthwhile.

---

From: Randall Stewart <[email-masked]>
Date: Tue, 17 Dec 2002 15:56:04 -0600

Jim

Comments below.

> until the larger PMTU is actually observed (usually it won't

Yes I do think that bundling of SCTP chunks is a logical way to
do PMTU. Has the protocol probes for a bigger PMTU it can combine
an increased number of chunks together to try to find a bigger
PMTU...

Now as far as a "null" chunk.. Nope.. not exactly.. however you can
(and some implementation DO) use the heartbeat chunk to probe for
PMTU changes... Basically the HB is whatever the sender wants in it
so a LOT of implementations will increase there HB size (which also
checks the back path as well since) to do this probe...

> not sure if that is how SCTP defines adoption indications.
Well.. the way it is currently defined, once an association is in
DDP/RDMA mode (with an adaption indication) the entire association is
IN this mode..maybe the wording needs to be clearified..

> information looks less than optimal.  

I think the idea here is we are leaving room for multiple messages
beside the stream reset... flexiility if you will.

Besides which we cannot send a "0" length message so we must send
at minimum a data chunk and 4 bytes... so if does not matter you get
a PPI in a data chunk no matter what you want... so even sending a
1 byte SCTP data message will result in

---------
sctp common-header
---------
data header
--------
1 byte
+
3 bytes pad
-------

aka 12 + 12 + 4 = 28 bytes

Also, with the current definition,
> thought should go into optimizing the combined header format.

Not sure why you think that you would lose alignment.. ALL data chunks
are 32 bit aligned.. if you mean in terms of DDP to RDMA.. I have
not looked closely at that but that becomes a DDP/RDMA draft issue
I think...

> else it should save a lot of time and reduce risk of rejection.

I know I too prefer the unordered option.. it makes a LOT of sense..

> may be applications where this is of significant value.

Not sure what you want then here... do you want to do both
ordered and unordered then?

> normal SCTP implementations.

The choice of 5.2 is not so clear to me.. mainly because you
can disable SCTP fragmentation in only one sense... You set
the socket option  SCTP_DIABLE_FRAGMENTS and then what will happen
any time you send a message bigger than the fragment point that
SCTP has declared you will get an error on the sending. If the
PMTU changes smaller than its fragment point the SCTP sender
will of course clear the DF bit (in v4) and resend the data
on any chunk that was fragmented to the larger size.

Note however that this will not stop an SCTP stack from combining
multiple chunks together to get close to the PMTU.. it only will
prevent the app from ever sending something bigger than SCTP's
fragmentation point (which is also settable with MAXSEG and will
almost always be min(user set MAXSEG, smallest-path-mtu))

> complexity is justified by the value.

I think it is worthwhile an allows for a lot of flexibility
to re-use the streams in an assocaition... requiring les
streams in any given assoc.

I agree we need to flesh this out more.. actually this draft
leaked out a bit early.. I had not planned to submit it until
we (the authors) got a few more edits and passes done.. but thats
ok..just remember it is not near cooked... and is still quite
a drafty draft in my mind...

>>                 [RDMA-Draft] MUST be followed.
Yeah the table format is off.. I will get it straightened out in the
next pass...

> Should be DDP rather than RDMA.  Formatting needs to be

yep.

-- 
Randall R. Stewart
[email-masked] 815-342-5222 (cell phone)

---

From: Randall Stewart <[email-masked]>
Date: Tue, 17 Dec 2002 16:01:42 -0600

Caitlin:

One comment.

> but doing so would require changes to both DDP and MPA. It didn't seem

Only problem with that is you would be attempting to subvert the
definition of a PPId. The PPId is reserved by IANA (you get them
like ports) and are suposed to specify what protocol (or sub-protocol)
is in a data message. So using it to indicate "its a pure DDP/RDMA
as defined in draft/rfc xxxx" or it is a "pure DDP adatpion negotiation
message (aka our stream reset) as defined in rfc xxxx" is also correct.

Saying I want to take over values 1 - N to indicate a queue value is not
in conformance with what the field was designed for nor do I think you
could get IANA to reserve a "block" of them.

The current draft matches what they shoudl be I think...

R