Subject: RE: [rddp] One vs Two CRCs

From: "Mukund, Shridhar" <[email-masked]>
Date: Thu, 5 Dec 2002 11:24:38 -0800

DDP draft authors,

I urge the authors of the DDP draft to make a few enhancements along the
lines 
of Digests. The issue of one vs. two CRCs has been well articulated for two
years 
by folks on the iSCSI reflector, by several folks on this reflector before
Atlanta IETF, 
by Jim and several others after Atlanta IETF.

The advantage argument for one CRC is minor and not compelling. On the other
hand, 
disadvantages of not having two CRCs are overwhelming. 

Some motivations:
   1. To the extent that digests are not end-to-end, they are that much
useless. 
       Digests should "logically" belong to RDMAP and ULPs such as iSCSI
when
       layered on top of DDP. (DDP can be the carrier and also the
calculator for 
       the end-to-end digest as discussed later).
   2. When RDMAP/DDP is off-loaded to hardware, it is important to consider
the
       hardware itself as part of the channel (from an error control
standpoint). This
       is in spite of the fact that the hardware may have various methods
for error 
       detection and correction internally. Granted, implementations will
vary 
       for good or bad reasons. That is probably why OEMs have RFQs to
select 
       which chip is good for them :-) All of the following should be
allowed:
	- An application wanting to skip data digest,  esp. if it is a s/w 
              implementation and is dealing with non-critical data.
            - An implementation that chooses (for good reasons) to checks
for
              data digest during DMA to the host memory.
	- An application that chooses to double check the data digest after
it has 
              reached the host memory for segments of mission-critical data.
            - An application that chooses to save the digest as its
error-detection-code.
            - ...
    3. Allow for the possibility of middleboxes that may want to re-route
the DDP
        stream, without having to sweat through data digest and exposing the
data to 
        errors by stripping out the data digest temporarily. Be it for
virtualization or 
        other creative applications that may make sense.
    4. ...

Requested Changes:
    1. A CRC32 digest for the DDP header. Since the DDP header is also the
carrier
       of RDMAP control, it doubles as the RDMAP control header digest.
	- MUST implement and MUST use; Can not afford to place data in the
wrong 
              place.
            - Aborting the connection on error may be OK, since DDP has no
abiltity to
              retry. Smaller fraction of the stream -> lower probability of
occurence.
              A good application should use ULP level completion status to
recover.
    2. A CRC32 digest for the DDP payload. This protects ULP message or ULP 
        payload. For example in the case of iSCSIoverDDP, the iSCSI header
will map
        to a ULP message and its data to ULP payload. Granted, the payload
may be
        segmented :-(
 	- MUST implement OPTIONAL use. It is perfectly alright to place bad
data 
              in the right location. (Some related arguments at the end)
            - Deliver to ULP with digest error. The application may choose
to ignore, retry 
              at the ULP level or abort connection. 
      3. When markers exist, they should not be covered by any of the
digests. As was 
          pointed out, markers are out-of-band and best kept that way.
      4. As Jim stated, there is no "compelling" need to add padding for
digest. It is not
          necessary for either hardware or software implementation.
Applications may 
          choose to be quad-byte aligned. If padding does exist, it should
not be covered 
          by digests. (arguable)

Side Effects:
    1. Get rid of digests from framimg layers.
    2. With thanks due to Jim, IFT may retire :-)
    2. The digest movement does not affect MPA style alignment. I am not
suggesting
        that I support alignment :-) iSCSI style markers for synchronization
        (not framing and alignment) is compelling though. 
    3.  If SCTP enthusiasts think appropriate, they may turn off data
digest. SCTP CRC is 
        in lieu of TCP checksum, not end-to-end application digest. SCTP CRC
may be good 
        enough esp. if SCTP is not off-loaded. (arguable)

Thanks.

-Shridhar Mukund

Optional to read comment:

*  It is OK to to place bad data in the right location as long as the data
is delivered with
   digest error.

   An argument was made that if the header arrives and is validated, data
will be moved to
   application buffers even if the data digest is not known. I do not see
any problems with 
   that.

   However in another argument, it was claimed that MPA (aligned frame) does
not place
   data when there is a data error. I think that argument is flawed. Given
that DDP may
   segment a ULP transaction into frames with independent CRCs, placing any
subset
   of those segments while one of them may have an error is same as placing
bad data
   in the right location from the ULP stand point.

_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: "Mukund, Shridhar" <[email-masked]>
Date: Thu, 5 Dec 2002 12:09:29 -0800

I aplogize for my badly formatted email. If I have not already
caused you pain, you may want to look at this one. Thanks.

-----Original Message-----
From: Mukund, Shridhar [mailto:[email-masked]]
Sent: Thursday, December 05, 2002 11:25 AM
To: [email-masked]
Subject: RE: [rddp] One vs Two CRCs 

DDP draft authors,

I urge the authors of the DDP draft to make a few enhancements 
along the lines of Digests. The issue of one vs. two CRCs has 
been well articulated for two years by folks on the iSCSI 
reflector, by several folks on this reflector before Atlanta 
IETF, by Jim and several others after Atlanta IETF.

The advantage argument for one CRC is minor and not compelling. 
On the other hand, disadvantages of not having two CRCs are 
overwhelming. 

Some motivations:
   1. To the extent that digests are not end-to-end, they are
      that much useless. 
      Digests should "logically" belong to RDMAP and ULPs such
      as iSCSI when layered on top of DDP. (DDP can be the 
      carrier and also the calculator for the end-to-end digest 
      as discussed later).
   2. When RDMAP/DDP is off-loaded to hardware, it is important 
      to consider the hardware itself as part of the channel 
      (from an error control standpoint). This is in spite of 
      the fact that the hardware may have various methods for 
      error detection and correction internally. Granted, 
      implementations will vary for good or bad reasons. That 
      is probably why OEMs have RFQs to select 
      which chip is good for them :-) All of the following 
      should be allowed:
	- An application wanting to skip data digest,  esp. if 
        it is a s/w implementation and is dealing with 
        non-critical data.
      - An implementation that chooses (for good reasons) to 
        checks for data digest during DMA to the host memory.
	- An application that chooses to double check the data 
        digest after it has reached the host memory for 
        segments of mission-critical data.
      - An application that chooses to save the digest as its
        error-detection-code.
      - ...
    3. Allow for the possibility of middleboxes that may want 
       to re-route the DDP stream, without having to sweat 
       through data digest and exposing the data to errors by 
       stripping out the data digest temporarily. Be it for
       virtualization or other creative applications that may 
       make sense.
    4. ...

Requested Changes:
    1. A CRC32 digest for the DDP header. Since the DDP header 
       is also the carrier of RDMAP control, it doubles as the 
       RDMAP control header digest.
	 - MUST implement and MUST use; Can not afford to place 
         data in the wrong place.
       - Aborting the connection on error may be OK, since DDP 
         has no abiltity to retry. Smaller fraction of the 
         stream -> lower probability of occurence.
         A good application should use ULP level completion 
         status to recover.
    2. A CRC32 digest for the DDP payload. This protects ULP 
       message or ULP payload. For example in the case of 
       iSCSIoverDDP, the iSCSI header will map to a ULP message 
       and its data to ULP payload. Granted, the payload may be
       segmented :-(
 	 - MUST implement OPTIONAL use. It is perfectly alright 
         to place bad data in the right location. (Some related 
         arguments at the end)
       - Deliver to ULP with digest error. The application may 
         choose to ignore, retry at the ULP level or abort 
         connection. 
      3. When markers exist, they should not be covered by any 
         of the digests. As was pointed out, markers are 
         out-of-band and best kept that way.
      4. As Jim stated, there is no "compelling" need to add 
         padding for digest. It is not necessary for either 
         hardware or software implementation. Applications may 
         choose to be quad-byte aligned. If padding does exist, 
         it should not be covered by digests. (arguable)

Side Effects:
    1. Get rid of digests from framimg layers.
    2. With thanks due to Jim, IFT may retire :-)
    2. The digest movement does not affect MPA style alignment. 
       I am not suggesting that I support alignment :-) iSCSI 
       style markers for synchronization(not framing and 
       alignment) is compelling though. 
    3. If SCTP enthusiasts think appropriate, they may turn 
       off data digest. SCTP CRC is in lieu of TCP checksum, 
       not end-to-end application digest. SCTP CRC may be good 
       enough esp. if SCTP is not off-loaded. (arguable)

Thanks.

-Shridhar Mukund

*  It is OK to to place bad data in the right location as long 
   as the data is delivered with digest error.

   An argument was made that if the header arrives and is 
   validated, data will be moved to application buffers even if 
   the data digest is not known. I do not see any problems with 
   that.

   However in another argument, it was claimed that MPA (aligned 
   frame) does not place data when there is a data error. I think 
   that argument is flawed. Given that DDP may segment a ULP 
   transaction into frames with independent CRCs, placing any 
   subset of those segments while one of them may have an error 
   is same as placing bad data in the right location from the 
   ULP stand point.

---

From: Michael Krause <[email-masked]>
Date: Mon, 09 Dec 2002 08:37:36 -0800

>       carrier and also the calculator for the end-to-end digest

Digests should be done at the lowest layer where they provide re-use 
opportunity for multiple ULP.  SCTP provides a 32-bit digest at the 
transport layer; MPA provides a 32-bit digest above TCP.  Both provide 
re-use opportunity thus benefit to both protocols.

>       (from an error control standpoint).

It is true that the local I/O and RNIC are both parts of the error control 
point of view.  This is why PCI-X 2.0 provides end-to-end ECC and PCI 
Express provides optional end-to-end CRC.  Within the RNIC, vendors will be 
required by many users to protect all critical data structures.  If these 
are sufficient to  meet you data integrity needs, then I see no reason to 
add complexity to the RMDA/DDP protocols to provide such additional checking.

>       This is in spite of

Interoperability is critical for these protocols to be successful.  Having 
more options is not good for interoperability.

Note: Software implementations for iSCSI deal with the digest issue so why 
would this be relaxed for RDMA?  Both are directly targeting remote memory 
for write operations (cannot do read operations without protocol off-load 
unless you want to take an interrupt to create a pseudo read) thus data 
integrity is not an option.

>         checks for data digest during DMA to the host memory.

As long as layering is preserved and the transport digest is validated 
prior to the DMA, then there is no issue here.

>         segments of mission-critical data.

Please explain what you envision here.  Is this a new interface that allows 
the application to extract the digest from the underlying implementation on 
a per FPDU basis?  Seems like a lot of overhead and complexity as not all 
RNIC will return such data?

>       - An application that chooses to save the digest as its

Where does it do this?

>         - MUST implement and MUST use; Can not afford to place

The single CRC already delivers thus there is no exposure today and its use 
is mandatory.  I'll note that earlier you state that it should be optional 
so this seems like a change in position within this particular e-mail string.

>        and its data to ULP payload. Granted, the payload may be

A single CRC covers both the header and the data.

>          to place bad data in the right location. (Some related

A single CRC only places good data at the right location which is a fairly 
strong ULP requirement from multiple ULP.

>          choose to ignore, retry at the ULP level or abort

If a single CRC provides all of the protection, why force the application 
to have an optional behavior for something that should not occur?

>          it should not be covered by digests. (arguable)

If it is simple to implement and low cost / free, then having a single 
method available to all is a good thing to specify.

>    the data digest is not known. I do not see any problems with

The current proposal is that data is only placed if it is known to be good 
which means the digest is known.

>    is same as placing bad data in the right location from the

Not quite true.  Only the frame where the data was bad is truly bad.  Data 
that has not been placed may be good and it is up to the application / ULP 
to determine whether it wants to find out or not - it would be responsible 
for understanding this discovery on its own.

Mike

---

From: Michael Krause <[email-masked]>
Date: Thu, 12 Dec 2002 18:29:42 -0800

At 06:00 PM 12/12/2002 -0800, Mukund, Shridhar wrote:

>Mike,
>
>I have to disagree with you on several accounts and on others we
>may be in agreement.
>
>BTW, has any thought been given to RDMA virtualization? Or atleast
>storage-over-RDMA virtualization? We sure do not want virtualization
>boxes to tamper with DATA digest, do we?
>
>... inline
>
> > >Some motivations:
>Given that we have two error detection methods at the the lower
>layers, the benefit of another method is maximized when it is further
>apart. If not some literature on reliable system design, please
>do take a look at the data gathered by:
>"When the Checksum and the data disagree", Jonathan Stone et. al.

I'm familiar with that paper.

>Reuse is good. In addition to reuse as suggested by you, I am
>suggesting that we allow it's reuse by the ULP as well. It comes at
>almost no additional cost. iSCSI provides end-to-end data digest.
>iSER (iSCSI over RDMA) will be handicapped in not being able to
>match that ability.

RDMA provides the same level of end-to-end that iSCSI does as an iSCSI HBA 
would terminate the protection just as a RNIC would.  Both technologies 
then rely upon the hardware and intervening infrastructure to maintain data 
integrity.

>Do you realize that you missed out PCI-64 and PCI-X 1.0? But that
>is not the point about end-to-end. The complexity of 2 CRCs relative
>to 1 CRC is *MINOR*.

I'm well aware of PCI and PCI-X which do support parity which isn't the 
best but at least provides one level of additional protection.

>If the digests are stripped out along with Ethernet CRC and TCP
>checksum close to the link layer, it is not covering the TOE
>process! This may not be a good design practice. I may not be
>in a position to comment on another implementation though.

Ethernet CRC and TCP checksum are stripped off today by Ethernet 
controllers as well as checksum off-load solutions.  Both are prevalent in 
all high-speed implementations.  TOE processing is no different in the host 
than in a RNIC - the data must be protected to the same extent in both 
domains (though not all hosts provide simple things such as ECC memory to 
avoid silent data corruption).  Adding in a second CRC provides no value.

>...
>Digests are optional in iSCSI. The scope of RDMA is much wider
>than that of iSCSI.

Digests are not optional in RDMA for just that reason.

>Atleast in the non-aligned case, you loose the markers and headers
>way back in the pipeline. 1 CRC may not have any meaning at the
>point of delivery to the host.
>
>I hope you are not assuming that all NIC/TOE vendors WILL optimize
>for the *aligned* case. We sure would not! That may or may not be
>bad business :-)
>
> >
>Passing the CRC to the ULP is not complex. I do agree that
>FPDU segmentation by DDP causes a concern. But that can be
>addressed.

Please explain the cost / complexity / benefit trade-offs.

>Good question. Maybe some of the storage box vendors may want to
>answer that :-) It's applicability is quite generic though.

While not a wire protocol issue per se it is something that would have to 
be required to allow interoperability of NIC / OSV / application.  Adding 
in options such as this is not conducive to achieving interoperability as 
well as the cost objectives many people desire for RNIC.

>Header digest MUST. Data digest OPTIONAL.

Disagree.  Simpler to have a single for both and always have it present so 
implementations have a single method they can rely upon always being present.

>...
> >From the application point of view, (assuming that only the
>application has the ability to retry) the entire *DDP message* is
>a *single* data-entity. As long as you are checking CRC at the
>*DDP segment* boundary, there is no way you can guarantee that
>the entire application data-entity is error free at placement time.
>
>Your claim is short. This is in no way different from the non-aligned
>assumption where bad data is temporarily placed in the right
>location, but caught before delivery.
>
>Your argument has value only if you are assuming retry at the DDP
>segment level. Are you? As far as the application is concerned,
>a bad bit anywhere is equally bad. I am assuming that the ULP has no
>clue about the DDP *segment* boundary. Digest error at the segment
>level has no meaning to the ULP.

Digest errors result in connection tear down.  RDMA cannot be responsible 
for a ULP targeting overlapping memory - that is a ULP issue.  RDMA states 
that all data placed is correct at the time it is delivered.  Any problems 
outside of this time period is the host or ULP problem and not RDMA to solve.

Mike