Subject: RE: [rddp] Security Draft issues - STag / PDs.

From: "Deleganes, Ellen M" <[email-masked]>
Date: Wed, 2 Jul 2003 11:28:55 -0700

My comments are embedded.

Ellen

-----Original Message-----
From: Caitlin Bestler [mailto:[email-masked]] 
Sent: Wednesday, July 02, 2003 10:55 AM
To: Culley, Paul
Cc: [email-masked]
Subject: RE: [rddp] Security Draft issues - STag / PDs.

On Wednesday, July 2, 2003, [email-masked] wrote:

>I think the issue would be clarified by taking some examples from the
>"verbs" draft (draft-hilland-rddp-verbs-00).
>

Well they would be, but they shouldn't be.

The verbs draft, by its own description, covers *one* style of
local interface that *may* be used. Explanations on the definition
of an STag, discussions of applicability and security should all
be valid across the full set of possible local interfaces.

<emd: Perhaps, but then it is likely that the security draft and
discussions of applicability will never be done. The full set of
possible local interfaces is probably infinite. However, it may be a
reasonable request to expand the discussions of applicability and
security to a larger set of "reasonable" (useful) local interfaces.>

>That draft says that each DDP stream (identified with a QP ID)
>is also always assigned a "Protection Domain" or PD on request
>of the ULP.  Some trusted component makes sure that the ULP is
>allowed to use that PD, either by assigning a new one, or
>otherwise checking that the ULP is allowed to reuse an already
>present one (possibly via a process ID).
>
>A QP is only in a single PD, but many QPs may be assigned to a PD,
>allowing certain cross QP remote access possibilities.
>
>When an STag is allocated, it is initially assigned to a PD as well on
>request of the ULP; the ULP is only allowed to assign a PD that it
>"owns" (this is to be enforced by a trusted and protected component,
>typically a module associated with the OS).
>
>When the STag is "registered" by the ULP to allow remote
>access, it may continue to use the PD (allowing access by any
>QP with the same PD), or be further "bound" to a specific QP
>(which thereafter limits access to the single QP referenced DDP
>stream).  The registration procedure also enforces that the ULP
>is accessing memory it owns (again, typically done with a
>trusted component, associated with the OS).  When the STag is
>invalidated, it reverts to PD level protection, allowing
>binding on any QP in the same PD the next time it is "Bound".
>

The above discussion mixes internal concepts with external ones.

"Protection Domain" is being used as both a "local memory access
authorization mechanism" and with its external meaning.

I prefer to work from the external definition only.

Hence the ULP creates one or more Protection Domains, enabling
that Protection Domain with some subset of the memory access
rights that it itself has.

Endpoints and STags can be created/associated with Protection Domains.

With these externally-oriented definitions, certain things become
obvious:

-- you should never place multiple STags or Endpoints in the same
   Protection Domain unless there is a ULP need to do so. The security
   draft seems to assume that a Protection Domain is matches the scope
   of the user process by default and provides complex advice on when
   to narrow it. With the external definition the advice is simple,
   don't make a Protection Domain any wider than the ULP requires.
-- The "stream identifier" is really just a degenerate subset of a
   default Protection Domain encompassing a single Endpoint and STags 
   that can only be used on that Endpoint.
<emd: I'm not sure what you mean by "default Protection Domain". A DDP
Stream should both be associated with a Protection Domain and a "stream
identifier". A valid programming model allows STags that can be used
across multiple streams and other STags that are valid only on that
single stream. It seems to be overly limiting to have the DDP Stream to
be associated only with one Protection ID, that is either its "stream
identifier" or the Protection Domain identifier, but not both.>
   
Now the local interface definitely should have a "Memory Regitration
Domain" or "Local Memory Region" that can encompass multiple Remote
Protection Domains. But that is a *local interface* optimization, it
allows the ULP to register memory *locally* once and enable it for
multiple Protection Domains. It simplifies coding logic, and makes
it easier for the RDMA stack to minimize address mapping resources.

But is it really relevant to the protocol, applicability or security?

I remain uncertain as to whether this is an "optional" feature that
is not being adequately described, or mere fumbling with terminology
because of presumptions about internal architectures.

I thought it had been settled that a remotely visible STag (the only
type that exists in the protocol) is defined by *one* "Protection ID"
and that *all* STags valid on the same DDP Stream would have to have
the *same* "Protection ID". Early language that implied that the
local interface would simply enable valid STag/Stream pairings
was specifically *rejected* because it implied that the RNIC
would have to search a list in order to validate, rather than
perform a single comparison.

<emd: This is not true, see above. *All* STags valid on the same DDP
Stream do not have to have the same scope (or same Protection ID as you
call it). Some could be valid across multiple streams, thus be
associated with the Protection Domain ID, while other STags may be valid
on only the single DDP Stream. I can imagine applications that have
would want to allow both models, depending on the memory that they are
exposing.>

If we're not doing *one*, we should revert to the simplicity of
"many". This "one except when maybe it might be two" is confusing.

<emd: I don't see a reason for the model where a DDP Stream is
associated with more than one Protection Domain Identifier. I can see
where one might want to associate multiple DDP Streams with the same
Protection Domain Identifier, but not vice versa. If there is a
programming model that requires this, please explain. Otherwise, it
complicates implementations much more than having the STag valid on a
Protection Domain Identifier OR the DDP Stream identifier.>

_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: Caitlin Bestler <[email-masked]>
Date: Wed, 2 Jul 2003 13:46:45 -0500

>Protection Domain Identifier OR the DDP Stream identifier.>

Allowing any set of Protection Domains to be valid for a DDP Stream
is the most general model. I can't really think of a strong reason
for doing it. It came up because the minimal statement that the
ULP simply enables STags with DDP Streams would have required it.

The whole point of the Protection Domain was to allow a *single*
check while allowing an STag to be valid on *multiple* DDP Streams.

Now the next issue is whether you are saying "having the STag be
valid on a Protection Domain Id XOR the DDP Stream ID" or if
you are specifying IOR.

With XOR, there is one check and the value being checked against
might be a Protection Domain ID or a Stream ID, you really have
a single tag of a parent type. As I have indicated previously,
all you have done is create a special name for a default Protection
Domain that the consumer does not have to explicitly create.

With IOR, you have a dual check.

Now if you have mistakenly set the Protection Domain to be the
scope of the application program rather than to the set of DDP
Streams associated with a single ULP session then the IOR might
look like a neat idea.

But if you narror the Protection Domain to the ULP required scope
to begin with, I do not see why you would want to further narrow
it for *some* STags. Multiple Stream sessions are somewhat of a
specialized application to begin with. Multiple Stream sessions
that are only kind-of-related to each other for some STags
strikes me as horrendously specialized -- and certainly not
worth doubling the complexity of the check for *all* STags.

Now it may be possible to minimize that performance penalty
by "zoning" the Stream IDs and the Protection Domain IDs,
but it all strikes me as unnessary complexity. An STag
belonging to one Protectin Scope, which is either a Protection
Domain (i.e., a set of DDP Streams), or a single DDP Stream,
is very simple, to understand, to implement and to use.
It also matches existing RDMA-aware APIs.

---

From: Caitlin Bestler <[email-masked]>
Date: Thu, 3 Jul 2003 12:02:13 -0500

>security to a larger set of "reasonable" (useful) local interfaces.>

Actually it is very easy. You simply confine your discussion of
Protection Domains to the degree that the end-to-end semantics
require them to exist.

Specifically:

A Protection Domain is a container for multiple DDP Streams
and STags. All STags within the Protection Domain are valid
upon all DDP Streams within the Protection Domain.

The Protection Domain SHOULD include no more DDP Streams than
are required by the ULP. An application SHOULD NOT include
unrelated DDP Streams merely because there is a local desire
to share a buffer pool.

The local interface SHOULD NOT place pressure on the ULP
to merge unrelated Protection Domains into a single Protection
Domain merely to reduce RNIC resource consumption. [i.e., there
needs to be someone to share mapping tables across multiple
Protection Domains when referencing the same shared buffer
region].

The local interface MUST allow the consumer to associate
an STag with remote read and/or write permission for any
contiguous subset of the consumer's address space.

The local interface MAY allow the consumre to associate
an STag with other sets of memory locations that might not
be the consumer's current memory map, might be contiguous, etc.
However, the local interface MUST NOT allow the consumer to
grant the RNIC any memory access privileges that the consumer
did not already have.

The local interface MAY require the consumer to preregister
memory to be used for reception of untagged messages. The
local interface MAY even choose to call the handle for this
type an "STag", if it really wants to confuse developers.
But it really SHOULD call it something like an LMR Context
or an L Key, so that everyone remembers that it is not the
same thing as a handle for a remotely accessible memory window
(that is called an STag, or an RMR Context / R Key).

---

From: Michael Krause <[email-masked]>
Date: Thu, 03 Jul 2003 11:17:09 -0700

>upon all DDP Streams within the Protection Domain.

Agreed.

>unrelated DDP Streams merely because there is a local desire

Conceptually true but not possible to enforce and in some cases, the desire 
to share a buffer region is quite strong and valid for the ULP in 
question.  I do not believe these should be included in any draft as 
strongly as you have shown above.

>Protection Domains when referencing the same shared buffer

Need to separate the v-to-p mappings which may be shared with the access 
controls which are the STag / PD which may or may not be shared.  Again, I 
don't see how what you propose is enforced thus it would be more of a 
recommendation than a SHOULD NOT.

>contiguous subset of the consumer's address space.

This would need to be constrained a bit.  It may be for a subset of the 
contiguous range - not necessarily all as one may not be able to pin-n-map 
the entire range for a given operation.

>(that is called an STag, or an RMR Context / R Key).

Registration is really about v-to-p mappings.  Doing it early prior to 
usage amortizes cost and thus reduces performance penalties.  I don't see a 
reason to change terms between a local and a remote STag.  The 
functionality is the same, i.e. access control and mappings so why be 
different.

Mike

---

From: Caitlin Bestler <[email-masked]>
Date: Thu, 3 Jul 2003 14:39:26 -0500

On Thursday, July 3, 2003, [email-masked] wrote:

>Conceptually true but not possible to enforce and in some
>cases, the desire to share a buffer region is quite strong and
>valid for the ULP in question.  I do not believe these should
>be included in any draft as strongly as you have shown above.
>
>Need to separate the v-to-p mappings which may be shared with
>the access controls which are the STag / PD which may or may
>not be shared.  Again, I don't see how what you propose is
>enforced thus it would be more of a recommendation than a
>SHOULD NOT.
>

My goal is simple.

There are appliations that naturally support a large number
of clients. They wish to pre-enable a large memory region
(in the LMR Context / Memory Region / L-Key sense) and obtain
a magic cookie from the OS that says "yes this application
really has access rights to this memory, and may bind Memory
Windows within it to obtain STags."

These applications then want to create seperate Protection
Domains for each client. And only enable each STag for that
specific Protection Domain.

Enabling an STag for one client may, on occassion, require
enabling it for more than one DDP Stream. The application
should be able to do this without being required to:

a)  duplicate memory registration resources, or
b)  expose the STag to *every* DDP Stream that is sharing
    the same buffer pool (or running in the same process).
    
That is, there is no correlation between sharing local resources
and sharing remote permissions. They are independent decisions.
It is reasonable, and can even be secure, for an application to use
one set of resources to support multiple clients. They merely
need to set up their STags properly, and probably zero out
buffers when first enabled for any given STag.

That is much simpler, and far less restrictive, advice to
given ULP developers who want to include multi-threaded
servers that support a large number of clients.

We need to clearly seperate the objects that control remote
access (the STags and Protection Domains that are referenced
in the DDP draft) from the objects that a given local interface
uses to register memory and validate user access to that memory.

Failure to do so can lead to confusion. I would contend that the
uneccesarily restrictive advice given in the security draft is
the result of exactly such confusion. And the authors of the
security draft are all highly familiar with the subject matter.
How much more easily would a developer using RDMA for the first
time be confused?

My interpretation of a SHOULD is that it is something a user
can reasonably expect to be present. There may be special 
considerations for not providing it, which ought to be explained.

You could downgrade the language to a recomendation, but I strongly
feel that applications designers need to be given the assurance that
they should design their interactions on the assumption that they
can enable an STag on the set of DDP Streams that are part of a
ULP session -- and make it no wider.

It would be a disservice, if they were left with the impression that
defining a Protection Domain to meet their requirements is a laudible
goal it is one that they will probably not be able to achieve, and
that instead they will have to choose between forcing all clients
using a single server to trust each other and launching a session
specific process for each client session.

>This would need to be constrained a bit.  It may be for a
>subset of the contiguous range - not necessarily all as one may
>not be able to pin-n-map the entire range for a given operation.
>
What I was intending to propose is that a *minimal* local
interface would at least be able to register a contiguous
range of the user's virtual memory map, and that there *may*
be a further ability to register non-contiguous portions
or to register raw buffers that are not mapped but which
the application has the appropriate permissions for anyway.