Subject: RE: [rddp] Problems with MPA Hello

From: [email-masked]
Date: Thu, 20 Feb 2003 17:30:13 -0500

Caitlin,

> Point 2: There is no such thing as an aligned TCP stream.
> There are only TCP streams that are currently alligned.
>
> Point 3: Whether or not the sender MPA aligns is at best
> a hint to the receiver.

Unfortunately, neither of these Points is consistent with
the buffering reduction discussion in Section 7 of Draft-Elzur
which is about a receiver allocating worst case resources
on the assumption that an aligned stream will stay aligned.

If your above two Points are correct, that discussion of
buffer optimization for aligned senders ought to be called
out in the MPA draft proper as something that an MPA receiver
SHOULD NOT do for robustness reasons.  Somehow, I don't
think that's what the MPA authors had in mind ... but,
if you can convince them to add such a "SHOULD NOT", that
would be progress.  I'll wait to see how that turns out
before venturing further into the other Points.

Thanks,
--David
----------------------------------------------------
David L. Black, Senior Technologist
EMC Corporation, 176 South St., Hopkinton, MA  01748
+1 (508) 293-7953             FAX: +1 (508) 293-7786
[email-masked]        Mobile: +1 (978) 394-7754
----------------------------------------------------

> -----Original Message-----
> From: Caitlin Bestler [mailto:[email-masked]]
> Sent: Thursday, February 20, 2003 1:41 PM
> To: [email-masked]
> Subject: [rddp] Problems with MPA Hello
> 
> 
> The introduction of an MPA Hello offers no actual solutions
> to any real problems. It merely complicates the protocols,
> and lengthens the minimum time for connection establishment.
> 
> 
> Point 1: MPA Mode can only be initiated by the ULP.
> 
> An MPA Hello cannot be used to initialize the MPA state.
> The most it can do is confirm the the peer ULP's intent.
> 
> Rationale: Under existing RFCs, an incoming TCP stream
> may consist of any sequence of bytes. There are no
> restrictions. Period. If MPA cannot "alter" TCP by
> specifying segmentation points that would have been legal
> anyway, then it *certainly* cannot place restrictions on
> what can be a TCP stream's payload.
> 
> A combined receicer that support Non-MPA and MPA over TCP
> MUST NOT enter MPA mode for any TCP connection prior to
> the point in the stream authorized by the local ULP.
> 
> 
> 
> 
> Point 2: There is no such thing as an aligned TCP stream.
> There are only TCP streams that are currently alligned.
> 
> The only information the the sending MPA layer has that
> the sending ULP lacks is whether the sending MPA will
> align the TCP segments it generates with FPDUs.
> 
> It CANNOT know that there will be no middlebox that will
> resegment the TCP stream for the simple reason that it
> CANNOT know what set of middleboxes will deliver the
> IP datagram.
> 
> Just as a reciever MUST be prepared to handle PMTU changes,
> it MUST be prepared to handle "unexpected" drops in
> alignment.
> 
> More fundamentally, I would submit that for MPA to claim
> forward knowledge of the behavior of an IP network is a
> violation of the most basic IP principles. If such network
> knowledge were available and legitimate to use, RDDP would
> benefit far more by simply ensuring that ALL DDP Segments
> were sized to conform with the *future* PMTU.
>  
> 
> 
> Point 3: Whether or not the sender MPA aligns is at best
> a hint to the receiver.
> 
> Niether the set of packets that will be accepted, nor the
> ULP required response to them, is affected by alignment.
> 
> A server MUST NOT drop a connection because it unexpectedly
> changes PMTU or drops alignment.
> 
> 
> 
> Point 4: There is no information for the sender MPA to add
> to an MPA Hello.
> 
> The sending MPA layer has no data that can be placed in an
> MPA Hello that was not already available to both ULPs.
> 
> 
> 
> 
> Point 5: The ULPs are already responsible for ULP matching.
> 
> As with every other IP protocol ever created, it is a ULP
> responsibility to know that its peer is complying wth the
> same protocol, and to take reasonable protections from both
> accidental connections and malicious attacks.
> 
> The use of MPA/TCP is already implicit in the identity of
> the ULP itself. There are no ULP protocols that work over
> *either* TCP or RDDP.
> 
> 
> 
> 
> Point 6: A confirming MPA Hello delays processing of
> other DDP segments received out-of-order.
> 
> If the purpose of an MPA Hello is to confirm the operative
> MPA mode *before* any MPA processing then all segments
> received out of order must be held for deferred processing.
> 
> 
> 
> 
> Point 7: The MPA Hello message wastes an entire TCP segment
> or violates DDP / LLP layering.
> 
> An MPA Hello message reduces the space available in the
> first LLP message. The DDP layer will be relying upon the
> normal PMTU when it creates DDP Segments. These will not
> fit in the same segment 
> 
> IF DDP layer reduces the size of the first segment to
> accomodate the MPA Hello then it is violating layering, by
> modifying its behavior based upon knowledge of the LLP.
> Unless we want to modify the LLP/DDP interface to report
> a suggested DDP segment size on a per request basis there
> is no way to accomodate this exception in an LLP neutral
> fashion.
> 
> For a *new* connection, especially a long haul one, wasting
> a single TCP segment is signifigant.
> 
> 
> 
> 
> 
> Caitlin Bestler
> http://asomi.com/CaitlinBestler/
> _______________________________________________
> rddp mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/rddp
> 
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: Caitlin Bestler <[email-masked]>
Date: Thu, 20 Feb 2003 17:56:00 -0600

On 2/20/03, [email-masked] wrote:

>the buffering reduction discussion in Section 7 of
>Draft-Elzur which is about a receiver allocating worst
>case resources on the assumption that an aligned stream
>will stay aligned.
>
>out in the MPA draft proper as something that an MPA
>receiver SHOULD NOT do for robustness reasons.  Somehow, I
>don't think that's what the MPA authors had in mind ...
>but, if you can convince them to add such a "SHOULD NOT",
>that would be progress.  I'll wait to see how that turns
>out before venturing further into the other Points.
>
This topic has already been thoroughly covered on the
reflector, but I'll go ahead and recap:

- Resegmentation is legal for TCP-aware middleboxes.

- The set of middleboxes that an IP datagram will traverse
should not be known by the ULP in advance. As that the ULP
has no means of physically preventing a network
re-configuration from occurring, it also cannot do so.

- Therefore it is possible to lose alignment "suddenly".
The ULP could be halfway through committing some file
or database changes when this happens.

- However it will be rare.

- An RNIC could make a statistical estimate of the
resources required to handle buffering caused by loss of
alignment. If those resources are exceeded, it will drop
the IP datagrams that required the exhausted resources.
This is no different than what any of the switches, routers
and other middleboxes between the two RNICs could do if
*their* statistical estimates are exceeded. Obviously,
just as with the middleboxes, the statistical estimate
SHOULD be heavily on the safe side. How safe is a market
issue, not an IETF issue.

- MPA/TCP is still a reliable transport protocol. It will
recover from IP datagrams being dropped due to resource
exhaustion. The validity of the ULP connection will not
be broken.

- An RNIC also has the option of "slow pathing" connections
via the host memory itself. Of course this is ultimately
just another form of buffering, and it can also be exhausted
so the point has not really changed. But it does emphasize
that requiring an RNIC to be robust in the face of lost
alignment DOES NOT require the addition of vast memory
resources on the RNIC itself.

Loss of alignment *will* have a throughput impact. So do
Ethernet collisions and Router-dropped packets. The point of
a  reliable transport layer is to be robust in the face of
such errors. If you value *guaranteed* speed over
reliability, then you should be using an unreliable
transport.

If we want to design something that only works in a Data
Center then why don't we call it IBoE (InfiniBand over
Ethernet). Since the goal is RDMA over IP, then we had best
accept the behavior of IP networks, complete with the need
to deal with statistical behavior, and design accordingly.

Requiring an MPA/TCP receiver to remain robust in the event
of alignment loss is what is required to truly be a reliable
IP transport protocol.

Caitlin Bestler

---

From: "Somesh Gupta" <[email-masked]>
Date: Thu, 20 Feb 2003 16:42:09 -0800

Caitlin,

Comment in line and then at the end.

> From: [email-masked] [mailto:[email-masked]]On Behalf Of
> Caitlin Bestler
> Sent: Thursday, February 20, 2003 3:56 PM
> To: [email-masked]; [email-masked]
> Subject: RE: [rddp] Problems with MPA Hello
> 
> 

  As David previously pointed out (I think), this is not so
  rare for software based senders. A look at the tcp_output
  function of bsd tcp (the code after send: label) will
  help clarify. Any time, data gets queued on the socket head
  because the sender is sending faster than tcp can send
  (congestion window is small due to slow start or congestion
   avoidance or the sender is just plain fast), the alignment
  will get broken. Of course this may not be easily perceived
  from traces where a single request/single response at a time
  model is being followed.

  However, with RDMA/DDP, there may be many messages outstanding
  at a time that are not a multiple of segment size and misalignment
  will happen unless the sender TCP is modified.

> Loss of alignment *will* have a throughput impact.

I think the issue is whether we will let implementations
decide this or the protocol decide this. I feel we should
let implementation decide this and design a framing protocol
that does not impose this.

So do
> to deal with statistical behavior, and design accordingly.

Agree.
> of alignment loss is what is required to truly be a reliable

  If you look at Jim William's proposal, it provides for a 
  TCP compliant framing proposal, where the implementor can
  decide whether to optimize for aligned senders (like in
  a data center), legacy senders (where migration is easier
  such as for iSCSI currently) or blend in both. I think
  that unless the protocol is targeted at the data center only,
  such an approach, which should be acceptable as 
  a standards track document, is better.

Somesh

---

From: Caitlin Bestler <[email-masked]>
Date: Thu, 20 Feb 2003 21:00:56 -0600

On 2/20/03, Somesh Gupta wrote:

> help clarify. Any time, data gets queued on the socket
> head because the sender is sending faster than tcp can
> send (congestion window is small due to slow start or
> congestion avoidance or the sender is just plain fast),
> the alignment will get broken. Of course this may not be
> easily perceived from traces where a single
> request/single response at a time model is being followed.
>
> However, with RDMA/DDP, there may be many messages
> outstanding at a time that are not a multiple of segment
> size and misalignment will happen unless the sender TCP
> is modified.
>
Let me clarify, it is the sudden loss of alignment that will
be rare. The overwhelming majority of unaligned connections
will be evidently so very quickly and/or they will only be
unaligned for relatively brief portions of their lifespan.

A software implementation based upon an existing TCP stack
that *attempts* to maintain alignment would be an example of
one that will *usually* maintain allignment.

One that makes no attempt will quickly be out of alignment,
and the receiver will quickly know not to expect alignment
on this connection.

I would anticipate that most RNICs will be engineered to
manage a large number of connetions, when most of those
connections are currently aligned.

Similarly, most RNICs will be engineered on the assumption
that the PMTU is not changing with every packet. It does not
mean they are not ready to deal with it when it does happen.

>let implementation decide this and design a framing
>protocol that does not impose this.
>

Agreed. But as Uri's analysis shows, there are substantial
advantages to processing of aligned FPDUs. Only a design
optimized for absolute guaranteed performance would not
end up working faster when faced with less work.

But I agree with the sentiment against it being an
imposition. The "will" was meant solely as a
prediction, not as a mandate.

>> 
>> Requiring an MPA/TCP receiver to remain robust in the
>> event of alignment loss is what is required to truly be
>> a reliable IP transport protocol.
>
> decide whether to optimize for aligned senders (like in a
> data center), legacy senders (where migration is easier
> that unless the protocol is targeted at the data center
> only, such an approach, which should be acceptable as a
> standards track document, is better.
>
I believe the current MPA draft also accomplishes this,
without the addition of a "MPA Hello" sequence or in any way
attempting to a priori guarantee alignment on certain
connections.

Jim's proposal also has a negotiation phase, which is my
primary objection. If there is to be a negotiation phase,
getting more done with it, as Jim's proposal does, makes
the pill slightly less bitter.

But my fundamental objection remains that there is no need
for a negotiation phase. It does not solve the problems it
is alleged to solve, it adds complexity, and has a non-zero
cost in terms of network traffic and latency.

Caitlin Bestler