Subject: RE: [rddp] Option or mandatory CRC in MPA - WG Chair speaks

From: [email-masked]
Date: Thu, 10 Apr 2003 21:14:35 -0400

> I find some of these arguments somewhat strange.  We have all entered
> into this effort to bring optimized processing with Remote Direct Data
> Placement, yet we continue to talk about, and give serious weight to,
> the argument concerning an all software implementation.  It seems that
> we might think about a complete software implementation, in passing,
> but this is NOT the goal of RDMA or this RDDP workgroup (in my opinion).

I need to say some fairly strong words on this.  That statement and
the implication that only hardware implementations need apply are at
direct odds with the discussion leading to WG rough consensus on the
ULPDU alignment with TCP segment issue, and the resulting rough consensus.
It is very clear that the discussion leading to that rough consensus:
- Emphasized the ability to run over existing unmodified TCP stacks.
- Emphasized the importance of fan-in scenarios in which a number
	of software clients communicated with a hardware-accelerated
	server.
I believe both of those represent the rough consensus of the RDDP
WG.  To the extent that they do not, we're back in the alignment
morass, as those were crucial to settling the alignment issue, so ...

I am putting the RDDP WG on notice that if this line of discussion
leads to a need to reopen the alignment issue, I may have no choice
but to conclude that the technical issues involved in RDDP over TCP
are of sufficient intractability that I need to recommend to the ADs
that the pending RDDP recharter should delete all TCP work.  

> Most of the applications that will be using the RDMA protocols already
> have existing TCP/IP implementations, and they are not enhanced or
> accelerated by a software implementation of RDMA.  If systems only
> have a standard NIC, then the applications should continue to use TCP/IP.

That needs a lot more discussion, as among other things it excludes
most NFS clients (by count of systems, not count of independently
developed implementations).  This reminds me of the early days of iSCSI
work when the "conventional wisdom" was that iSCSI was going to need
accelerated hardware - at the moment software iSCSI implementations
are coming in with acceptable performance, albeit not as good as could
be achieved in hardware.

It is fine to discuss whether the CRC should be optional, but that
discussion MUST respect the WG rough consensus that *finally* closed
the alignment issue if any progress is going to be made.  Consideration
of all-software clients and their requirements is necessitated by the
current state of the RDDP WG, unless people want to reopen the alignment
issue (see above if you want to do that).

Thanks,
--David
----------------------------------------------------
David L. Black, Senior Technologist
EMC Corporation, 176 South St., Hopkinton, MA  01748
+1 (508) 293-7953             FAX: +1 (508) 293-7786
[email-masked]        Mobile: +1 (978) 394-7754
----------------------------------------------------
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: John Hufferd <[email-masked]>
Date: Thu, 10 Apr 2003 19:58:53 -0700

David,
I think you read to much into this.  It had nothing to do with the 
alignment issues.  That discussion had valid points independent of the 
arguments being made about CRCs.  We have folks debating the CPU overhead 
with CRC,  and trying to add things which OPTIMIZE software implementation 
for handling CRCs (like not doing it).  That was the issue and what was 
talked about in this note, NOT alignment!  I know that SW will exist, but 
trying to optimize for them is another issue.
.
.
John L. Hufferd
Senior Technical Staff Member (STSM)
IBM/System Group, San Jose CA
Main Office: (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688
Home Office: (408) 997-6136, Cell: (408) 499-9702
Internet Address: [email-masked]

[email-masked]
Sent by: [email-masked]
04/10/2003 06:14 PM

        To:     John Hufferd/San Jose/IBM@IBMUS, [email-masked]
        cc: 
        Subject:        RE: [rddp] Option or mandatory CRC in MPA - WG Chair speaks

                 of software clients communicated with a 
hardware-accelerated

Thanks,
--David

---

From: "Culley, Paul" <[email-masked]>
Date: Fri, 11 Apr 2003 09:12:37 -0500

I agree that the WG consensus was to "The RDDP protocols must allow
implementations over unmodified TCP/IP stacks." (as quoted from your
"Alignment consensus attempt" email.

I was unable to locate any minutes describing the fan-in scenarios that
would lead to a conclusion of "consensus".  That said, I do believe that
such scenarios will exist.

In both cases I am not quite so sure that the group agreed to
"emphasize" operation over Unmodified SW stacks; rather, I for one, only
think these should be enabled.  Perhaps I am reading too much into your
choice of words.

In my mind "enabling" these scenarios, while important, should not mean
that we go to great lengths to optimize or reduce overhead on the
software side at the expense of making the overall protocol less robust,
or adding multiple options that applications will have to sort through
and all concerned will have to test.

I believe that most of the scenarios where clients might be implemented
in software will also be able to afford the relatively small extra
overhead that use of CRCs requires.  These clients will already be
incurring the overhead of copies on receive, so only CPU time (not cache
dirties or extra memory cycles) should be added in reasonably good code.
The transmit side may incur be a bit more overhead, but I note that many
client applications are asymmetrical anyway, they receive more than they
send.

Also, I firmly believe that we cannot completely drop CRCs, that
essentially all customers will be quite nervous about allowing memory
access by DDP headers that have not been checked by something better
than a checksum.  So there will be that overhead anyway.  I think that
checking the remaining user data will be important enough to the
majority of customers that putting the option in for the minority that
don't care that much is just not worth it.

Paul R. Culley
HP Senior Fellow
281-514-5543

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] 
Sent: Thursday, April 10, 2003 8:15 PM
To: [email-masked]; [email-masked]
Subject: RE: [rddp] Option or mandatory CRC in MPA - WG Chair speaks

> but this is NOT the goal of RDMA or this RDDP workgroup (in my 

I need to say some fairly strong words on this.  That statement and the
implication that only hardware implementations need apply are at direct
odds with the discussion leading to WG rough consensus on the ULPDU
alignment with TCP segment issue, and the resulting rough consensus. It
is very clear that the discussion leading to that rough consensus:
- Emphasized the ability to run over existing unmodified TCP stacks.
- Emphasized the importance of fan-in scenarios in which a number
	of software clients communicated with a hardware-accelerated
	server.
I believe both of those represent the rough consensus of the RDDP WG.
To the extent that they do not, we're back in the alignment morass, as
those were crucial to settling the alignment issue, so ...

I am putting the RDDP WG on notice that if this line of discussion leads
to a need to reopen the alignment issue, I may have no choice but to
conclude that the technical issues involved in RDDP over TCP are of
sufficient intractability that I need to recommend to the ADs that the
pending RDDP recharter should delete all TCP work.  

> have a standard NIC, then the applications should continue to use 

That needs a lot more discussion, as among other things it excludes most
NFS clients (by count of systems, not count of independently developed
implementations).  This reminds me of the early days of iSCSI work when
the "conventional wisdom" was that iSCSI was going to need accelerated
hardware - at the moment software iSCSI implementations are coming in
with acceptable performance, albeit not as good as could be achieved in
hardware.

It is fine to discuss whether the CRC should be optional, but that
discussion MUST respect the WG rough consensus that *finally* closed the
alignment issue if any progress is going to be made.  Consideration of
all-software clients and their requirements is necessitated by the
current state of the RDDP WG, unless people want to reopen the alignment
issue (see above if you want to do that).

Thanks,
--David

---

From: [email-masked]
Date: Fri, 11 Apr 2003 13:13:16 -0400

> such scenarios will exist.

Good - I hope I don't have to spend the time to extract the relevant
supporting discussions from the WG email list archives.  

> think these should be enabled.  Perhaps I am reading too much into your

Perhaps - my use of the word "emphasize" was solely with respect to
issues that were important in establishing rough consensus on
the alignment issue.  Unmodified software stacks are important, but
my post was not about any notion of "emphasizing" them to the exclusion
of other implementation structures. 

> software side

That's ok - somewhere in the range from "enabling" to "go to great
lengths" (including the endpoints) there should be an engineering
tradeoff that will yield a protocol that is reasonably implementable
in both hardware and software.  This is in stark contrast to John's
original call to ignore considerations for software implementations:

	It seems that we might think about a complete software
	implementation, in passing, but this is NOT the goal of
	RDMA or this RDDP workgroup (in my opinion).

Just in case it's not clear - while John is entitled to his opinion,
the immediately above statement is simply wrong about the IETF RDDP
WG, as it is at odds with prior WG rough consensus.  Attempting to
change the position of the WG to match the quoted statement carries
significant risk of unraveling the rough consensus on the alignment
issue, which may have dramatic consequences for the scope of the WG's
efforts.  That's the clue-by-four - I would advise against inviting
the use of anything larger ...

Paul's email goes on to touch on a couple of other issues that
I'm going to take up under different subject lines.

Thanks,
--David

---

From: John Hufferd <[email-masked]>
Date: Fri, 11 Apr 2003 10:43:29 -0700

David,
There was no intent in my note, to attempt to invalidate previous 
agreements.  In fact it was to accept the previous agreement and move on 
-- without attempting to ALSO apply extensive work to optimization the 
software performance issues -- that I was attempting to motivate.   That 
is quite different than dealing with the issues of unmodified TCP Stacks, 
that applied to the alignment issue.

So, to the extent that my words lead you to think I was attempting to 
revolt from the previous agreements, which I support, I regret that.  It 
was NOT my intention.
.
.
John L. Hufferd
Senior Technical Staff Member (STSM)
IBM/System Group, San Jose CA
Main Office: (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688
Home Office: (408) 997-6136, Cell: (408) 499-9702
Internet Address: [email-masked]

[email-masked]
04/11/2003 10:13 AM

        To:     [email-masked], [email-masked]
        cc: 
        Subject:        RE: [rddp] Option or mandatory CRC in MPA - WG Chair speaks

Thanks,
--David