Subject: RE: [rddp] STag Invalidation Question

From: "Culley, Paul" <[email-masked]>
Date: Fri, 7 Feb 2003 13:21:46 -0600

I agree with Caitlin, there are lots of applications that want to open
up a potentially large buffer for access by a remote agent; that remote
agent will often want to perform multiple access, perhaps even involving
writes and reads to that buffer before it gets invalidated.  The
Send+Invalidate allows the remote side to perform the invalidate when
all such access are completed (usually when the ULP operation has been
satisfied and when returning a status to the local ULP).  

Although not specified in the RDMAP specification, the local ULP should
be informed when or if the Invalidate was done, so it can cross check
that it has been done.

Most folks in the original author team thought that the difference in
time/security between getting a "one-shot" invalidation, and
invalidating by the ULP when the ULP has been told that the buffers are
no longer being used remotely, to be small enough that requiring the
"one-shot" functionality was not worth it.

Paul R. Culley
HP Senior Fellow
281-514-5543

-----Original Message-----
From: Caitlin Bestler [mailto:[email-masked]] 
Sent: Friday, February 07, 2003 11:08 AM
To: Williams, Jim; [email-masked]
Subject: Re: [rddp] STag Invalidation Question

On 2/7/03, Williams, Jim wrote:

>
>I am sure this issue has been discussed and debated before although 
>perhaps not in this forum.  I would be interested in hearing some of 
>the thoughts here, though.
>
>PROPOSAL:  STags are always invalidated immediately on
>           on completion of RDDP writes.  This is done
>           before delivery of the data to the application.
>           Therefore all tagged buffers are "write once"
>           and must be re-registered before they can
>           be written again.
>

There are numerous reasons for wanting to re-use STags that relate to
the presumption that binding them for each operation will be too
expensive.

I fully agree that those reasons are insufficient. Implementations
SHOULD assure that binding and invalidating an STag for each RDMA Write
is a truly valid option. No ULP should refrain from doing so for fear of
performance penalties.

Numerous algorithms developed for dual-ported memory are applicable to
RDMA and use of persistent STags.

The primary example for me remains Variable Bit Rate (VBR) delivery. The
Data Sink advertises a single large buffer intended to accept data over
an extended period of time. Sends will be sent at timed intervals to
confirm how much data has been placed (and allowing it to be deliverd).

Because the amount of data transferred per completion is variable, there
is no way to pre-feed future STags. The starting address for pipeline
stage N begins *after* stage N-1, and this would not have been known
when a distinct STag would have had to have been created.

Another example would be a SAN or NAS server that has been asked to
write 100 MB from a client buffer specified with an STag and TO.

Unless it can dedicate a 100 MB buffer (unlikely), the
server will be using RDMA Reads to fetch chunks matching
half of its buffer size (perhaps 1 MB), and then writing
from those buffer to disk. (Two buffers wouuld be used at
the minimum so that one can be pre-loaded while the other
is written to disk).

These have to be distinct RDMA Reads, so that a completion
is generated for each portion. The completion would trigger
the actual write to disk.

It is outside the scope of the IETF, but I would certainly recomend that
applications SHOULD use one-shot STags, and that support for
invalidation SHOULD be efficient enough so that no ULP would avoid
one-shot STags for performance reasons.

The applications that have need to use persistent STags have valid
reasons of the type needed to override a SHOULD. They are advanced
techniques that should not be selected by default.

Caitlin Bestler
http://asomi.com/CaitlinBestler/
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: [email-masked]
Date: Fri, 7 Feb 2003 17:25:37 -0700

Caitlin,

There are reasons for wanting to do multiple RDMA operations to an advertised buffer that have nothing to do with the expense of binding to the STag. These reasons often have to do with letting the client and the server have independent control of the amount of buffering they use to support transfers.

Your write of a file from a client to a server is one example of this. Another example is the case where a client ULP requests its server ULP to write a file. The client sends an STag in the request to advertise the buffer for the file. The server may not have all the data ready for the transfer at once and the client shouldn't have to know that detail in order to set up its buffer. When the server ULP has data ready to transfer to the client, it will start an RDMA Write for that data. Note that an RDMAP receiving RDMA Writes does not deliver them the ULP. Whether the request takes one RDMA Write or many is transparent to the client ULP.

When all the RDMA Writes necessary to transfer the data have been initiated, the server ULP can send an RDMA Send with Invalidate message to invalidate the STag and carry a ULP PDU that informs the client ULP that the request has been fulfilled. This send may also have other information that the ULP will need in order to read the data such as the length of data that was actually written if the response was not a fixed size.

Since all prior RDMA Writes must have completed before the RDMA Send message is delivered to the ULP, receipt of the Send ensures that the data is in the advertised buffer.

Invalidating STags on RDMA Writes as Jim suggests would be incompatible with the operation above and it is entirely unnecessary. Since RDMA Writes are not surfaced to the ULP, it is the Send which tells the ULP that it can use the Data. Therefore, the Send with Invalidate is the tool that supplies the control that Jim wants.

Regards,
Pat

-----Original Message-----
From: Caitlin Bestler [mailto:[email-masked]]
Sent: Friday, February 07, 2003 9:08 AM
To: Williams, Jim; [email-masked]
Subject: Re: [rddp] STag Invalidation Question

On 2/7/03, Williams, Jim wrote:

>
>I am sure this issue has been discussed and debated before
>although perhaps not in this forum.  I would be interested
>in hearing some of the thoughts here, though.
>

There are numerous reasons for wanting to re-use STags that
relate to the presumption that binding them for each
operation will be too expensive.

I fully agree that those reasons are insufficient.
Implementations SHOULD assure that binding and invalidating
an STag for each RDMA Write is a truly valid option. No ULP
should refrain from doing so for fear of performance
penalties.

Numerous algorithms developed for dual-ported memory are
applicable to RDMA and use of persistent STags.

The primary example for me remains Variable Bit Rate (VBR)
delivery. The Data Sink advertises a single large buffer
intended to accept data over an extended period of time.
Sends will be sent at timed intervals to confirm how much
data has been placed (and allowing it to be deliverd).

Because the amount of data transferred per completion is
variable, there is no way to pre-feed future STags. The
starting address for pipeline stage N begins *after* stage
N-1, and this would not have been known when a distinct STag
would have had to have been created.

Another example would be a SAN or NAS server that has been
asked to write 100 MB from a client buffer specified with an
STag and TO.

It is outside the scope of the IETF, but I would certainly
recomend that applications SHOULD use one-shot STags, and
that support for invalidation SHOULD be efficient enough
so that no ULP would avoid one-shot STags for performance
reasons.

The applications that have need to use persistent STags have
valid reasons of the type needed to override a SHOULD. They
are advanced techniques that should not be selected by
default.

Caitlin Bestler

---

From: John Hufferd <[email-masked]>
Date: Fri, 7 Feb 2003 17:56:11 -0800

I completely agree with Pat. She is describing the way a storage controller
works with a host.  It would make no sense for the storage controller to
get additional STags, from the host, for each segment of I/O it is sending
to the Host (The size and number of segments depends on the
Buffering/Caching approach taken by the Storage Controller).  Plus the host
will not know what buffering and rotational delay, etc.  is being handled
by the Storage Controller so it will not know how many STags to send the
Storage Controller for any specific I/O request.  This may cause the
Storage Controller to ask the host to send additional STags, in the middle
of the I/O transfer, so that it can finish the requested I/O.  (Not a very
good design.)

.
.
.
John L. Hufferd
Senior Technical Staff Member (STSM)
IBM/Systems Group, San Jose  CA
Main Office (408) 256-0403, Tie:276-0403, eFax: (408) 904-4688
Home Office: (408) 997-6136, Cell: (408) 499-9702
Internet Address: [email-masked]

                      pat_thaler@agilen                                                                                        
                      t.com                    To:       [email-masked], [email-masked], [email-masked]                
                      Sent by:                 cc:                                                                             
                      [email-masked]        Subject:  RE: [rddp] STag Invalidation Question                                 
                      rg                                                                                                       

                      02/07/2003 04:25                                                                                         
                      PM                                                                                                       

Caitlin,

There are reasons for wanting to do multiple RDMA operations to an
advertised buffer that have nothing to do with the expense of binding to
the STag. These reasons often have to do with letting the client and the
server have independent control of the amount of buffering they use to
support transfers.

Your write of a file from a client to a server is one example of this.
Another example is the case where a client ULP requests its server ULP to
write a file. The client sends an STag in the request to advertise the
buffer for the file. The server may not have all the data ready for the
transfer at once and the client shouldn't have to know that detail in order
to set up its buffer. When the server ULP has data ready to transfer to the
client, it will start an RDMA Write for that data. Note that an RDMAP
receiving RDMA Writes does not deliver them the ULP. Whether the request
takes one RDMA Write or many is transparent to the client ULP.

When all the RDMA Writes necessary to transfer the data have been
initiated, the server ULP can send an RDMA Send with Invalidate message to
invalidate the STag and carry a ULP PDU that informs the client ULP that
the request has been fulfilled. This send may also have other information
that the ULP will need in order to read the data such as the length of data
that was actually written if the response was not a fixed size.

Since all prior RDMA Writes must have completed before the RDMA Send
message is delivered to the ULP, receipt of the Send ensures that the data
is in the advertised buffer.

Invalidating STags on RDMA Writes as Jim suggests would be incompatible
with the operation above and it is entirely unnecessary. Since RDMA Writes
are not surfaced to the ULP, it is the Send which tells the ULP that it can
use the Data. Therefore, the Send with Invalidate is the tool that supplies
the control that Jim wants.

Regards,
Pat

-----Original Message-----
From: Caitlin Bestler [mailto:[email-masked]]
Sent: Friday, February 07, 2003 9:08 AM
To: Williams, Jim; [email-masked]
Subject: Re: [rddp] STag Invalidation Question

Caitlin Bestler

---

From: "Williams, Jim" <[email-masked]>
Date: Mon, 10 Feb 2003 15:08:56 -0500

pat_thaler wrote:

> Your write of a file from a client to a server is one example 
> of this. Another example is the case where a client ULP 
> requests its server ULP to write a file. The client sends an 
> STag in the request to advertise the buffer for the file. The 
> server may not have all the data ready for the transfer at 
> once and the client shouldn't have to know that detail in 
> order to set up its buffer. When the server ULP has data 
> ready to transfer to the client, it will start an RDMA Write 
> for that data. Note that an RDMAP receiving RDMA Writes does 
> not deliver them the ULP. Whether the request takes one RDMA 
> Write or many is transparent to the client ULP.

You make a good point.   I think I agree with this.
Need to think a bit more about the whole problem.

 - Jim

---

From: [email-masked]
Date: Mon, 17 Feb 2003 15:35:20 -0500

WG Chair hat off ...

> Need to think a bit more about the whole problem.

I don't see a major issue with this sort of interface paradigm:
- The receiving ULP wants the entire lump, and doesn't care how
	many pieces it arrives in.
The receiving ULP isn't going to do anything until all the data
arrives, and can arrange for the STag to be invalid at that point.

Caitlin's time-limited delivery scenario is somewhat different:
- The receiving ULP definitely wants to do something with the
	data while leaving the STag valid for more to arrive.
One possibility that would allow the receiving ULP to protect
itself would be to have some way to change the STag range without
affecting its validity (e.g., a Send arrives saying "40K made it
through in the last time period before I sent this", so the
receiving ULP moves the lower bound of the STag up by 40K
before starting to use the data).

This would need to be considered vs. advising the ULP to use an
alternative design of alternating (or round-robining) among a
buffers/STags, where the buffer switch is time-driven (e.g., and
a Send indicates how much data was sent) to determine
how important it is to be able to change the region of memory
described by an STag without invalidating the STag.

Thanks,
--David
----------------------------------------------------
David L. Black, Senior Technologist
EMC Corporation, 176 South St., Hopkinton, MA  01748
+1 (508) 293-7953             FAX: +1 (508) 293-7786
[email-masked]        Mobile: +1 (978) 394-7754
----------------------------------------------------