Subject: [rddp] Protection Domain Definition

From: Caitlin Bestler <[email-masked]>
Date: Thu, 3 Jul 2003 16:51:12 -0500

I believe the root of the problem with the discussion on STags and
Protectin Domain can be directly traced to an incorrect definition
of Protection Domain used in the proposed verbs.

The protocol defines a Protection Domain as follows:

>   Protection Domain (PD) - A Mechanism used to associate a DDP Stream 
>       and an STag. Under this mechanism, the use of an STag is valid 
>       on a DDP Stream if the STag has the same Protection Domain 
>       Identifier (PD ID) as the DDP Stream.

While the verbs draft defines something else called a "Protectin Domain"

>   Protection Domain (PD) - A mechanism for tracking the association of 
>       Queue Pairs, Memory Windows, and Memory Regions. PDs are 
>       intended to be set by a Privileged Consumer to provide 
>       protection of one process from accessing another's memory 
>       through the use of the RNIC.  

The latter, I'll call them Local Protection Domains, are clearly wider
than the protocols definition of a Protection Domain.

This leads to two problems:

a) A mistaken belief that servers that share local resources are sharing
   remote access rights.
b) A belief that it is necessary to have a mechanism to narrow the scope
   of an STag to a single DDP Stream because a Protection Domain is too
   wide. Well, if its a Local Protection Domain, of course its too wide.
   
If Protection Domains can be created independently of Local Protection
Domains (subject only to the general rule that the Local Interface MUST
NOT grant the RNIC any authority that the caller did not already have)
then the ULP can already tailor an STag to be valid only for the set
of DDP Stremas that are part of a single session.

Secondly, if make the mistake of widening the Protection Domain to the
scope implied by the Local Protection Domain then we still do not have
a solution to allow the ULP to make an STag valid *only* on the DDP
Streams that are part of a *single* session. A multi-threaded server
could have 10,000 sessions, 30,000 DDP Streams and a *single* Local
Protection Domain.

If I try *real* hard, I can come up with a theoretical scenario
where I can have 10,000 sessions, each with 3 DDP Streams, and
there are some STags that are to be used only on a single stream.
But how common is it? And if we deal with that, do we also have
to deal with applications that need to allow the STag to be
valid on 2 out of 3 Streams within a Protection Domain?

Let's keep things simple. Allowing Protection Domain *or*
QP ID carries costs:

-- The bind operation must distinquish whether the STag is
   being bound to the entire Protection Domain or to one Stream.
   
-- The MW query must indicate whether the STag is bound to the
   QP ID or to the PD ID.
   
-- The semantics of a peer invalidate are unclear. 

-- There must be a constraint against a QP ID being destroyed
   and recreated while an STag is bound to it. Either it must
   be prevented, or the STag must be nullified. Either requires
   maintenance of a reverse link and/or claim counts.
   
And for what benefit?

If you mistake a Protection Domain for a Local Protection Domain
it can appear very compelling. There will be lots of servers
supporting 10,000 clients who only want an STag to be valid
upon a single DDP Stream.

But those can be handled by simply creating a Protection Domain
for each session -- which will almost always be a single DDP Stream.
Indeed many of the classic problems solved by extra connections
will no longer require extra DDP Streams under RDMA. FTP data
transfers are one example.

My evaluation is that if Protection Domains are *properly* scoped,
whether or not the local interface has its own concept of Local
Protection Domain, that the number of applications that will need
to control which STag is valid on which subset of the DDP STreams
associated with a single session will be *very* slight -- so
slight that it is not worth *any* additional complexity.

The verbs draft should be updated to seperate concerns on local
memory protection from ULP / protocol issues. The security draft
should be updated based upon the assumption that the local interface
will support Protection Domains as intended in the DDP draft.

_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp