Subject: RE: FW: [rddp] One vs Two CRCs

From: "Mukund, Shridhar" <[email-masked]>
Date: Thu, 12 Dec 2002 18:00:39 -0800

Mike,

I have to disagree with you on several accounts and on others we 
may be in agreement.

BTW, has any thought been given to RDMA virtualization? Or atleast
storage-over-RDMA virtualization? We sure do not want virtualization
boxes to tamper with DATA digest, do we?

... inline

> >Some motivations:
> >    1. To the extent that digests are not end-to-end, they are
> >       that much useless.
> >       Digests should "logically" belong to RDMAP and ULPs such
> >       as iSCSI when layered on top of DDP. (DDP can be the
> >       carrier and also the calculator for the end-to-end digest
> >       as discussed later).
> 
> Digests should be done at the lowest layer where they provide re-use 
> opportunity for multiple ULP.  SCTP provides a 32-bit digest at the 
> transport layer; MPA provides a 32-bit digest above TCP.  
> Both provide 
> re-use opportunity thus benefit to both protocols.

Given that we have two error detection methods at the the lower
layers, the benefit of another method is maximized when it is further 
apart. If not some literature on reliable system design, please 
do take a look at the data gathered by:
"When the Checksum and the data disagree", Jonathan Stone et. al.

Reuse is good. In addition to reuse as suggested by you, I am
suggesting that we allow it's reuse by the ULP as well. It comes at 
almost no additional cost. iSCSI provides end-to-end data digest. 
iSER (iSCSI over RDMA) will be handicapped in not being able to 
match that ability. 

> 
> >    2. When RDMAP/DDP is off-loaded to hardware, it is important
> >       to consider the hardware itself as part of the channel
> >       (from an error control standpoint).
> 
> It is true that the local I/O and RNIC are both parts of the 
> error control 
> point of view.  This is why PCI-X 2.0 provides end-to-end ECC and PCI 
> Express provides optional end-to-end CRC.  Within the RNIC, 
> vendors will be 
> required by many users to protect all critical data 
> structures.  If these 
> are sufficient to  meet you data integrity needs, then I see 
> no reason to 
> add complexity to the RMDA/DDP protocols to provide such 
> additional checking.

Do you realize that you missed out PCI-64 and PCI-X 1.0? But that
is not the point about end-to-end. The complexity of 2 CRCs relative 
to 1 CRC is *MINOR*. 

If the digests are stripped out along with Ethernet CRC and TCP
checksum close to the link layer, it is not covering the TOE 
process! This may not be a good design practice. I may not be
in a position to comment on another implementation though.

...

> 
> Note: Software implementations for iSCSI deal with the digest 
> issue so why 
> would this be relaxed for RDMA?  Both are directly targeting 
> remote memory 
> for write operations (cannot do read operations without 
> protocol off-load 
> unless you want to take an interrupt to create a pseudo read) 
> thus data 
> integrity is not an option.

Digests are optional in iSCSI. The scope of RDMA is much wider 
than that of iSCSI. 

> 
> >       - An implementation that chooses (for good reasons) to
> >         checks for data digest during DMA to the host memory.
> 
> As long as layering is preserved and the transport digest is 
> validated 
> prior to the DMA, then there is no issue here.

Atleast in the non-aligned case, you loose the markers and headers
way back in the pipeline. 1 CRC may not have any meaning at the
point of delivery to the host.

I hope you are not assuming that all NIC/TOE vendors WILL optimize
for the *aligned* case. We sure would not! That may or may not be 
bad business :-)

> 
> >         - An application that chooses to double check the data
> >         digest after it has reached the host memory for
> >         segments of mission-critical data.
> 
> Please explain what you envision here.  Is this a new 
> interface that allows 
> the application to extract the digest from the underlying 
> implementation on 
> a per FPDU basis?  Seems like a lot of overhead and 
> complexity as not all 
> RNIC will return such data?

Passing the CRC to the ULP is not complex. I do agree that
FPDU segmentation by DDP causes a concern. But that can be 
addressed.

> 
> >       - An application that chooses to save the digest as its
> >         error-detection-code.
> 
> Where does it do this?

Good question. Maybe some of the storage box vendors may want to 
answer that :-) It's applicability is quite generic though.

> 
> >       - ...
> >     3. Allow for the possibility of middleboxes that may want
> >        to re-route the DDP stream, without having to sweat
> >        through data digest and exposing the data to errors by
> >        stripping out the data digest temporarily. Be it for
> >        virtualization or other creative applications that may
> >        make sense.
> >     4. ...
> >
> >Requested Changes:
> >     1. A CRC32 digest for the DDP header. Since the DDP header
> >        is also the carrier of RDMAP control, it doubles as the
> >        RDMAP control header digest.
> >         - MUST implement and MUST use; Can not afford to place
> >          data in the wrong place.
> 
> The single CRC already delivers thus there is no exposure 
> today and its use 
> is mandatory.  I'll note that earlier you state that it 
> should be optional 
> so this seems like a change in position within this 
> particular e-mail string.

Header digest MUST. Data digest OPTIONAL.

...

> 
> >         - MUST implement OPTIONAL use. It is perfectly alright
> >          to place bad data in the right location. (Some related
> >          arguments at the end)
> 
> A single CRC only places good data at the right location 
> which is a fairly 
> strong ULP requirement from multiple ULP.

From the application point of view, (assuming that only the 
application has the ability to retry) the entire *DDP message* is
a *single* data-entity. As long as you are checking CRC at the 
*DDP segment* boundary, there is no way you can guarantee that 
the entire application data-entity is error free at placement time.

Your claim is short. This is in no way different from the non-aligned
assumption where bad data is temporarily placed in the right 
location, but caught before delivery.

Your argument has value only if you are assuming retry at the DDP 
segment level. Are you? As far as the application is concerned, 
a bad bit anywhere is equally bad. I am assuming that the ULP has no 
clue about the DDP *segment* boundary. Digest error at the segment 
level has no meaning to the ULP.

...

Regards,

-Shridhar Mukund
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: Caitlin Bestler <[email-masked]>
Date: Thu, 12 Dec 2002 21:23:53 -0600

There is a problem with providing ULP level data digests on
Tagged messages -- the receiving side cannot distinquish
between A 2K tagged message that was segmented into two 1K
segments, and two 1K tagged messages that happen to be
adjacent.

[BTW, this makes the whole TO order requirement doubly
absurd. If we accepted the argument that this requirement be
included, there would be no way to detect a deviation.]

ULPs are already free to define their own protection
mechanisms. I don't see the reason for adding an additional
complexity that is optional to use.

Yes, things can go wrong between the NIC and the host. But
the NIC and host are part of the same system. They have to
have their own mechanisms for making transfers reliable.
If they are not there, what would be the point of detecting
an error in transferring ULP data but *not* detecting an
error in transferring a page mapping table?

If host/NIC communications are not reliable then DDP will
not be able to define an feature, optional or mandatory,
that will solve the problem.

Caitlin Bestler
http://asomi.com/CaitlinBestler/
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: Michael Krause <[email-masked]>
Date: Fri, 13 Dec 2002 06:39:03 -0800

>that will solve the problem.

Strongly agree with all of the above.

Mike