Subject: RE: [rddp] Some Minimal end-to-end Buffer Rules Required

From: Michael Krause <[email-masked]>
Date: Thu, 17 Jul 2003 06:00:45 -0700

At 04:23 AM 7/17/2003 -0500, Culley, Paul wrote:
> > -----Original Message-----
> > From: Caitlin Bestler [mailto:[email-masked]]
> > Sent: Wednesday, July 16, 2003 9:10 AM
> > To: John Hufferd
> > Cc: [email-masked]
> > Subject: Re: [rddp] Some Minimal end-to-end Buffer Rules Required
> >
> >
>Deleted...
> >
> > Then these applications do not work over InfiniBand? You have
> > a credit limit over IB RC, and it is even less flexible than
> > what you can do with iWARP.
> >
>Actually these applications (that occasionally send more messages than
>normally credited) could work quite well on IB because IB supports
>automatic flow control for the RC mode receive queue buffers.  In the
>"normal" case, the flow control is not invoked, everything flows
>smoothly.  In the exceptional case when too many messages need to be
>sent, IB backs up the messages at the sender (in the Send Queue).

Exactly.  IB was designed with a send credit scheme that would allow an 
application to post as many Send operations as it desired (up to its SQ 
limit) knowing the that underlying interconnect would insure that no more 
than what had been actually credited by the receiver would be sent at any 
given time.  A simple example: application has a lot of work to do, gets a 
time slice, posts a lot of work, completes it time slice.  It does not know 
if the remote connection is resource thin or rich thus it relies on the 
credit scheme to allow the remote to advertise its credit as a function of 
its resource capabilities for this particular connection.  Please note that 
all of this is focused on data center distances which many viewed as to why 
it was a practical idea; consensus on this list was the credits were not 
good due to supporting WAN distances, analysis of existing ULP which 
already use a credit scheme at the ULP level, etc.

Mike

_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: Caitlin Bestler <[email-masked]>
Date: Thu, 17 Jul 2003 09:11:36 -0500

On Thursday, July 17, 2003, [email-masked] wrote:

>
>Exactly.  IB was designed with a send credit scheme that would
>allow an application to post as many Send operations as it
>desired (up to its SQ limit) knowing the that underlying
>interconnect would insure that no more than what had been
>actually credited by the receiver would be sent at any given
>time.  A simple example: application has a lot of work to do,
>gets a time slice, posts a lot of work, completes it time
>slice.  It does not know if the remote connection is resource
>thin or rich thus it relies on the credit scheme to allow the
>remote to advertise its credit as a function of its resource
>capabilities for this particular connection.  Please note that
>all of this is focused on data center distances which many
>viewed as to why it was a practical idea; consensus on this
>list was the credits were not good due to supporting WAN
>distances, analysis of existing ULP which already use a credit
>scheme at the ULP level, etc.
>
Two quibbles:

1) This discussion actually occurred during the yahoogroups
era, so it technically wasn't "this list".

2) The WAN implications were not merely distance, but the
possibility of intermediate termination of the TCP stream.
Renato's early drafts for iWARP had IB-like send credits,
but they relied upon the LLP ack as feedback.

TCP middleboxes are allowed to terminate and re-originate a
stream, making a TCP ack meaningless to the iWARP sender.
Given that limitation, it was agreed that adding an
explicit iWARP ack was not a viable solution. Therefore
the credits had to be deleted, which was reflected in
the first IETF drafts.