Subject: RE: [rddp] Options and Markers

From: "Williams, Jim" <[email-masked]>
Date: Tue, 15 Apr 2003 15:39:51 -0400

> From: Caitlin Bestler [mailto:[email-masked]]
> [...]
> That leave the real issue being assessing which is the 
> greater imposition:
> 
> -- forcing senders to support both marker and no-marker mode.
> -- forcing memory-rich receivers to implement markers.
> 
> The primary virtue of the latter is that it avoids options. 
> Allowing recievers
> to mandate "no markers" is clearly superior for pure-software 
> RDDP stacks
> written on top of sockets.

If you will forgive a bit of oversimplification, I view
the problem as follows.

If markers are optional to implement, then there are two
possible outcomes:

    A:  Markers turn out to be valuable, and all implementations
        will either support them or be non-competitive.

    B:  Markers turn out NOT to be valuable, and will not
        be implemented.

If markers are mandatory to implement (regardless of whether
optional or mandatory to use) then there are two possible
outcomes:

    C:  Markers turn out to be valuable and are always used.

    D:  Markers turn out NOT to be valuable, but must be 
        implemented anyway for compliance.

So A and C are virtually equivalent outcomes, but B is clearly
preferable to D.  D is clearly worse if markers are also 
mandatory to use.  Overall, my preference is to make
markers optional to implement unless or until their 
value is demonstrated through actual implementations.

_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: Caitlin Bestler <[email-masked]>
Date: Tue, 15 Apr 2003 18:39:25 -0500

> value is demonstrated through actual implementations.

You are assuming that markers being "valuable" is a universal thing.
The point I was making is that markers are extermely valuable to some
receivers (those with low memory resources) and of no value to those
without that restriction (indeed, the markers becomes a slight nuisance).

I believe there is an overwhelming case that some recievers require markers,
and that failure to provide them will have a severe impact on their
performance. As pointed out, the buffering requirements scale by the
maximum in flight allowance for each connection.

---

From: Jeffrey Mogul <[email-masked]>
Date: Tue, 15 Apr 2003 17:22:29 -0700

After watching this debate from the cheap seats for a while,
and realizing that I don't have a strong technical preference
either for or against either markers and CRCs, Jim's case
analysis inspired me to comment.

My sense of the debate is that:

    for all X in the set {Markers, CRCs}

	1. Some people want X
	2. Some people don't want X
	3. X has some overhead that might lead to suboptimal performance
	4. X is not hard to implement

	and the choices for the RDDP-WG are:

	(a) Don't put X in the protocol
	(b) Make X mandatory
	(c) Make X fully optional
	(d) Make X mandatory to implement, but optional to use

But rather than to address this as a technical question, I'd like
to address it as a procedural question.

First, our goal is to achieve "genuine consensus" (or to give
up).  Unanimity is not required by IETF procedures.  So I think
we should consider which choices are most likely to lead to
consensus.

   This means that choices (a) and (b) are a problem, because
   they lead to continued debate (and perhaps re-opening the
   debate at future stages of the IETF process.)

Second, my experience from several past IETF standards (Path MTU
Discovery, HTTP/1.1) is that interoperability suffers unless the
specification leaves very little wiggle room.  Unlike Jim, I am
not optimistic that the marketplace will save us from products
that don't support "valuable" features whose implementation is
optional.  (For example, in a corporate network with 50,000 nodes
and hundreds or thousands of routers, it only takes one or two
routers that black-hole PMTUD messages to force nearly everyone
to turn off their end-host PMTUD support.)

   This means that choice (c) is a problem, because if anyone
   does want X, then we'll always be fixing interoperability bugs
   in deployed systems.

Third, the IETF process does allow changes to the protocol design
between the Proposed Standard and Draft Standard stages.  But in
practice, it's very hard to add a feature here, because the IESG
seems to be suspicious of new features without sufficient
testing.  So they may require us to go through the Proposed
Standard stage again (with all the delay that implies) if we try
to add something.

    This means that choice (a) is a problem if some people really
    do want X.

So I lean towards choice (d): Make X mandatory to implement, but
optional to use.  This will make fans of X happy, will add some
implementation overhead for opponents of X, and will annoy people
who don't want any options.

So, are options really that bad?  I don't think so.  First of
all, if X is mandatory to implement but optional to use, and the
receiver gets to choose, then you can make this work without
extra round-trips by having the sender start the connection using
the option, and then let the receiver tell the sender to turn it
off.

Second, my guess is that any solution to the "prove MPA doesn't
get confused with unmodified TCP" requirement will provide us
with a convenient place to pass some options.  (This is a
requirement because the IESG says it is a requirement; it doesn't
matter what my personal technical judgement is, because the IESG
has the last say.  I've been there before.)

Third, I'm really nervous about publishing *any* protocol spec
that doesn't include a version number that the WG can change
relatively easily.  Especially one where people are clearly
tempted to deploy systems based on "Proposed Standard"
specifications.  We had this problem with HTTP/1.1, and by not
making the decision to increment the version number between
Proposed and Draft, we created massive and permanent
interoperability problems.  If you have no version number at all,
you're equally screwed.

At this point, I don't believe MPA carries an explicit version
number, and I think this is a disaster waiting to happen.  If we
fix that, we again are likely to have a place to put options,
too.

-Jeff

---

From: Caitlin Bestler <[email-masked]>
Date: Thu, 17 Apr 2003 11:36:42 -0500

On 4/15/03, Jeffrey Mogul wrote:

>So, are options really that bad?  I don't think so.  First
>of all, if X is mandatory to implement but optional to
>use, and the receiver gets to choose, then you can make
>this work without extra round-trips by having the sender
>start the connection using the option, and then let the
>receiver tell the sender to turn it off.
>
Starting markers in the middle of the data would be very
complex. I agree that this can be made to work without 
an extra round-trip, but there is still a requirement 
that the markers be present (or absent) from the very
beginning. This does make the first segment special,
but that is a given (if we accept that some form of
protocol marker has been mandated by the powers-that-be).
 
>Second, my guess is that any solution to the "prove MPA
>doesn't get confused with unmodified TCP" requirement will
>provide us with a convenient place to pass some options. 
>(This is a requirement because the IESG says it is a
>requirement; it doesn't matter what my personal technical
>judgement is, because the IESG has the last say.  I've
>been there before.)

The two are definitely related.

Personally, I do not find the "same ULP different de facto
protocol but using the same number" argument compelling.
There is no guard seperating HTTP over TLS-SSL/TCP from
HTTP over TCP. But I will concede that there is at least
some merit to the proposal, and it can be argued that
TLS *should* have been required to do something similiar
but that it is too late.

For the record, though, I still believe this requirement
should be stated to tsvwg and debated there.

Assuming that it is something that RDDP will end up with,
however, it then makes sense to communicate other mode
information at the same time.

I believe optional markers is clearly a case where there
is benefit to using this packet that we have to pay for
anyway (i.e., "free") to set a connection specific option.

I believe the debate has established the following points:

--- There is an implementation strategy ("cut through"
    or "low memory") where omitting markers would
    substantially increase the amount of buffering
    required for an integrated RDDP/MPA/TCP stack.
    
    Not allowing such receivers to mandate markers in
    streams sent to them would have a serious impact
    on their performance.
    
--- There are also implementation strategies based
    upon cleaner layering where the TCP layer has
    sufficient buffering to still provide "single
    touch" placement by simply deferring placement
    until the gap TCP segments arrive.
    
    Having to deal with markers in such environments
    provides no benefits, and is at least a minor
    performance penalty. I believe the remaing dispute
    is whether this is a *severe* penalty.
    
--- It is possible to specify an opening "mode establishing"
    packet which will ensure that each side sends markers
    to match the expectation/demand of the receiver without
    requiring an additional round-trip before real data
    can be exchanged.
    
I remain convinced that the cost of markers to the "software
only" / "buffer rich" / "layered" solutions does not qualify
as "severe" and that it would not justify the creation of a
mode packet.

But, if a mode packet is going to exist *anyway*, why not
enable this simplification for receivers that have valid
reasons to avoid the use of markers? The extra byte required
to set this option up is surely justified. There is the
extra QA cost for senders, which must support both modes,
but I for one am convinced that the case for allowing some
receivers to work without markers has been made
sufficiently.

Caitlin Bestler
http://asomi.com/CaitlinBestler/
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: Jeffrey Mogul <[email-masked]>
Date: Fri, 18 Apr 2003 17:03:12 -0700

Caitlin Bestler wrote:

Note that, in the paragraph you quoted, I didn't propose *starting*
markers in the middle of the data.  Ending them in the middle
seems a lot simpler.  For example, one could have a marker value
that means "this is the last marker on this connection."  Doesn't
the MPA spec include some RESERVED bits in the market format?

Note: I don't really understand all 35 pages of the MPA spec,
so maybe I'm confused.

At any rate, my goal is not to re-engineer the specs, it's to
see if we can find some sort of acceptable compromise that
will bring working-group consensus closer.

-Jeff

---

From: Caitlin Bestler <[email-masked]>
Date: Sun, 20 Apr 2003 00:46:48 -0500

On 4/18/03, Jeffrey Mogul wrote:

>   complex. I agree that this can be made to work without > 
>   beginning. This does make the first segment special, but
>   that is a given (if we accept that some form of protocol
>   marker has been mandated by the powers-that-be).
>
>Note that, in the paragraph you quoted, I didn't propose
>*starting* markers in the middle of the data.  Ending them
>in the middle seems a lot simpler.  For example, one could
>have a marker value that means "this is the last marker on
>this connection."  Doesn't the MPA spec include some
>RESERVED bits in the market format?
>

Changing the marker mode, rather on or off, requires a
round-trip.

The problem is that any change implies that there is a
point in the stream where the change occurs. There also
has to be some point in the stream at or before that where
the intent to make the change is given.

You have ot guarantee that the notice has been received,
and processed, before any segments in the new mode arrive.
The only two methods that I can think of require an explicit
ack or giving notice further in advance than the
connection's window allows (hence the notice and first data
in the new mode cannot both be in flight at the same time).

The latter only works if the notice is handled by MPA
itself. If it is a ULP message, then a ULP ack is needed.
Google some of the earlier discussions on *entering* RDDP
mode, its pretty much the same problem.

Caitlin Bestler

---

From: "Vadim Makhervaks" <[email-masked]>
Date: Sun, 20 Apr 2003 09:16:17 +0300

Absolutely agree. To change the mode one need to stop the traffic, make
sure that the change-mode-notification has been received and accepted, and
only then proceed with the data transmission. Otherwise, think about
out-of-order received segments ...

Vadim

|---------+---------------------------->
|         |           Caitlin Bestler  |
|         |           <[email-masked]> |
|         |           Sent by:         |
|         |           [email-masked]|
|         |           rg               |
|         |                            |
|         |                            |
|         |           20/04/03 08:46 AM|
|---------+---------------------------->
  >-------------------------------------------------------------------------------------------------------------------------------|
  |                                                                                                                               |
  |       To:       Jeffrey Mogul <[email-masked]>                                                                             |
  |       cc:       [email-masked]                                                                                                 |
  |       Subject:  Re: [rddp] Options and Markers                                                                                |
  >-------------------------------------------------------------------------------------------------------------------------------|

Caitlin Bestler

---

From: "Somesh Gupta" <[email-masked]>
Date: Sun, 20 Apr 2003 20:16:13 -0700

The RDDP protocol suite will require a reasonable number of
exchanges between the two systems before data can be transferred
in a manner that makes use of the RDDP protocol.

1. Authentication of end points is for all practical purposes,
a very strong requirement.
2. Exchange of buffer information is needed for direct data
placement
3. Storage connections are long lived in any case.
4. IPC connections - looking at SDP as an example - have a 
 number of exchanges for setup.

Given this, one approach should be to see how the cost of
MPA parameter exchange can be minimized instead of using
that as a reason for not having options.

I understand testing as a concern but let me give an
example for the iSCSI protocol.

The immediate/unsolicited data option in iSCSI is an optional
feature. Making it mandatory overlooks the fact that most
existing targets do not support that capability. Not
having it in the protocol overlooks the fact that it removes
write latency. Having it as an option allows for faster
deployment of iSCSI, while in the future allowing for this
as a performance improvement. So overall we had the following
choices:

1. Immediate data on or off
2. Unsolicited data on or off
3. The amount of immediate data
4. The amount of unsolicited data

Of course it is one of the examples of option creep. But
we have gotten over it and people are ready to deploy
products.

In RDMA we are talking about a handful of options on features
where I feel we have reasonable difference of opinion
on various aspects of those features.

Somesh

---

From: Caitlin Bestler <[email-masked]>
Date: Mon, 21 Apr 2003 09:58:41 -0500

On 4/20/03, Somesh Gupta wrote:

>The RDDP protocol suite will require a reasonable number
>of exchanges between the two systems before data can be
>transferred in a manner that makes use of the RDDP
>protocol.
>
RDDP is not an application layer protocol, it rides just
above the traditional transport layer and essentially
functions as a transport layer.

As with any transport, it should avoid making presumptions
about what the ULP will do or not do.

Any distinctions should relate to the inherent nature of
RDDP, not to presumptions about its use. For example, RDDP
might not handle a given interchange well. That is not a
problem if the given interchange does not meet the
characteristics that RDDP was intended to optimize.

Ultimately, the defining characteristic of ULPs that benefit
from RDDP is that there is a relatively large amount of data
that is exchanged for each ULP signifigant event.

That characteristic does not support your assertions.

In particular, the assertion offered would not apply to
a ULP that used RDMA to allow anonymous clients to fetch
files or other named assets (such as HTTP over RDDP).

>1. Authentication of end points is for all practical
>purposes, a very strong requirement.

Authentication is seldom required for fetching content.
A single request could be sent which identified the resource
requested and supplied the advertised buffer where it was to
be deposited. The RDMA transfers and a single reply could
complete the session.

>2. Exchange of buffer information is needed for direct data

DAFS is already an example of a ULP that includes the buffer
advertisement in the request. No additional exchanges are
required. DAFS is an authenticated file system. It chooses
to set up sessions to authenticate clients, but that is part
of that ULP, it is not intrinsic to its use of RDMA.

>3. Storage connections are long lived in any case.

Classic storage is only one use of RDMA.

HTTP connections are highly optimizable using RDMA and are
only as long lived as required to fetch a single page.

> number of exchanges for setup.

Again, that is a single ULP. In fact it is one where the two
SDP peers are managing a complex shared buffering resource.

There is obviously a minimum duration of a connection to be
worth designating RDMA in the first place. There simply
isn't a signifigant penalty in being forced to serialize
data transfer unless you are transferring at least a certain
amount of data. Nobody worries about reordering buffers for
a single 16 KB file.

>that as a reason for not having options.

Given that MPA supports bundling, including a MPA parameter
block at the start of MPA-mode is definitely acceptable.

Valid objections remain as to the QA cost of each option,
but the network impact of one-way MPA paramter notices that
can be bundled with actual data is clearly acceptable.

MPA parameters that require acknowledgement, however, add a
round trip delay. There are a large class of transfers that
could have benefitted from RDMA transfer, but would not when
an extra round trip was added to the start of the exchange.

With high speed connections, a *lot* of data could have been
transferred in the time it takes for an extra round trip.
And yes, the first round trip won't be very fast anyway. But
we've also delayed attempting to send large amounts of data
and expanding the transmit window. Ultimately, a round-trip
delay is a round-trip delay.

Caitlin Bestler

---

From: Michael Krause <[email-masked]>
Date: Mon, 21 Apr 2003 07:07:41 -0700

>a very strong requirement.

There is no additional requirement beyond what is required for normal 
connection / stream establishment - anything else is a ULP / application 
issue outside the scope of the protocol.

>2. Exchange of buffer information is needed for direct data

Ideally, a ULP mapping will simply map the existing messages exchange on 
top of the protocol thus there is no increase in the number of messages 
exchanged.

>3. Storage connections are long lived in any case.

The protocol is not focused on any one market segment - there is nothing to 
preclude using RDMA for shorter lived connections.

>  number of exchanges for setup.

There is a hello / ack message exchange similar to an iSCSI login thus 
there is an additional round trip at most.

>that as a reason for not having options.

Keeping as many options out of the "data mover" layer of an implementation 
is a good thing irrespective of whether the ULP above it are smart in their 
execution and thus minimization of options / number of messages exchanged / 
etc.

>I understand testing as a concern but let me give an example for the iSCSI 
>protocol.
>
>The immediate/unsolicited data option in iSCSI is an optional feature. 
>Making it mandatory overlooks the fact that most existing targets do not 
>support that capability. Not having it in the protocol overlooks the fact 
>that it removes write latency. Having it as an option allows for faster 
>deployment of iSCSI, while in the future allowing for this as a 
>performance improvement. So overall we had the following choices:
>
>Of course it is one of the examples of option creep. But we have gotten 
>over it and people are ready to deploy products.

While an interesting, option creep has and will always be a bad thing to 
simply accept and move onward.  I've been involved in other technologies 
where too many options have been nearly the death of the technology itself 
- it wasn't until we all agreed on what the options were thus making them 
no longer options did people actually get the technology implemented in any 
credible form.

>In RDMA we are talking about a handful of options on features where I feel 
>we have reasonable difference of opinion on various aspects of those features.

My preference is to not have options aside from resource depths wherever 
possible.  Keeping options out of the wire protocols is a good thing for 
all.   To date, I have only seen a couple of people claim the marker option 
is important while many others claim otherwise.  Are there additional 
people who believe one way or another that should be heard?

Mike