Subject: [rddp] RDDP Buffer Issues

From: "Williams, Jim" <[email-masked]>
Date: Thu, 14 Nov 2002 18:14:00 -0500

RDDP protocol purports to allow direct out of order placement
of data in user buffers while preserving in order semantics
on data delivery.  This is done while layered on top of an 
ordered transport.

Effective DDP turns the user buffer temporarily into the
transport reassembly buffer, potentially eliminating
the data copy from the transport buffer to the user buffer.

In order for this to work and to preserve the ordering
semantics of the underlying transport, the following rules
MUST be followed.

1.  The DDP client must not peek or poke
    the memory locations which will hold data
    while data is being placed.  Only after the
    data is delivered to the
    application may the application touch it.

2.  After DDP has delivered data to an application,
    DDP must not place new data on top of the old
    data without insuring that the application is
    done peeking and poking the affected memory.

3.  There must never be two RDDP writes targeting
    the same memory location within a given transport
    window, or else reordering of the placement  
    of these will visible.  This is unacceptable.

I believe it is clear that violating any of these
rules violates the ordering semantics of the transport
layer, and also results in non-deterministic behavior
as seen by the application.

I am concerned that the current proposal for DDP is
too sloppy on insuring that these rules won't be violated.
It is not broken outright, in that a properly behaving
client will result in the rules being followed. But
is has the following weaknesses which I would
propose should be fixed.

A.  The ordering semantics are too heavily dependent on
    the client behavior.  The rules that the client
    must follow are not as simple or explicit as they
    should be.

B.  Proper ordering semantics is dependent on the remote
    end of a connection behaving properly.  It is desirable
    that one end of a connection should be able to insure
    that its transport layer semantics are not violated
    without having to rely in subtle ways on the 
    behavior of remote higher protocol layers.

C.  Following rule 2 requires that the remote end to
    not overwrite the data in question, but it also
    depends on the local end interpreting DDP headers
    correctly even though the transport layer has
    not delivered and may never deliver them.  The
    DDP layer independently verifies them by means
    of its own CRC.  In practice, the should work.
    But in a formal academic sense, this is a 
    clear violation of layering that should be avoided
    in protocol design.

==========================================

One possible solution to the above is
to define a clear buffer ownership policy.  Registering
a buffer with the DDP layer also passes ownership of that
buffer.  The application is not allowed to peek or poke
any buffer that the DDP layer owns.

The DDP layer delivering data constitutes transferring 
ownership of the containing buffer back to the application
layer. At this point DDP may not write any more data in that 
buffer.

The DDP layer must check that there is no overlap among
any registered buffers, and must check that each buffer
is written only once before ownership is given up.
Any attempt by the remote end to write an un-owned buffer
should be detected as a protocol error and cause
the connection to terminate in error.

Effectively, this means that all buffers are one-shot, 
and must all be non-overlapping.  This is a definite
change to the proposed functionality.

But the client rules become trivially simple:  don't
touch a buffer that you don't own.  This frees the
DDP layer to use buffers that it owns any way it
wants without needing to worry about the correctness
of what it's doing to those buffers (until, of course,
it's ready to deliver data and pass buffer ownership).

This is not the only way to approach this problem,
but will I hope start a discussion.

_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: Caitlin Bestler <[email-masked]>
Date: Thu, 14 Nov 2002 18:51:46 -0600

On 11/14/02, Williams, Jim wrote:

>RDDP protocol purports to allow direct out of order
>placement of data in user buffers while preserving in
>order semantics on data delivery.  This is done while
>layered on top of an ordered transport.
>

As is, DDP must have completed placements prior to
completion delivery. DDP MUST NOT alter user data except in
response to untagged or tagged placements.

So what you are proposing is a ULP restriction, or a
requirement that the DDP actively prevent the ULP from being
stupid. With limited exceptions, I do not believe this is an
appropriate requirement. More specifics in later responses.

>    of these will visible.  This is unacceptable.

This is a general statement about distributed systems
design. As Mike Krause will be sure to comment unless he
read this first, its a pure ULP issue.

Enforcement of this type of rule by the DDP is very
difficult, worse it is prone to false positives.

For example: a client could be connected to each member of a
storage cluster. It sends a request to any member of th
cluster, and in a pattern internal to the cluster they all
deliver portions of the requested data in parallel. The
client knows the entire buffer is present when *all* of the
connections have replied, but does not know which of them
delivered what portion of the data.

Short of byte-granularity scoreboarding their would be no
way for the the DDP layer to detect that this application
was actually performing correctly.

>    must follow are not as simple or explicit as they

I doubt anyone would have objections to including many rules
on how applications MUST behave if they are using DDP
properly. But almost nobody will want to burden what should
be a thin DDP layer with dynamically enforcing these rules.

>    behavior of remote higher protocol layers.

This is one argument for invalidating tags locally. I had
earlier objected to doing so remotely, because I felt it was
more properly undre local control. Valid scenarios were
presented where remote validation was needed. The consensus
at the time was that local invalidation could be implemented
by any local interface.

A case could be made that this MUST be a local interface
capability, as that it is the only other way to prevent
malicious retro modifications of a target buffer would
require the receiving ULP to postpone examinging the data
until *after* it had invalidated the STag itself.

Attaching an optional STag Invalidate to a pre-posted read
is one way to solve this. In my opinion it would be an
acceptable solution to the problem you raise.

I might have philosophical quibbles as to whether this
merits a MUST qualification, rather than a SHOULD. Is it
really the IETF's role to tell someone how much they should
spend on their NIC to be secure? It could be argued that
this is like the police telling you that you MUST lock your
door at night, rather than suggesing that you SHOULD do so.

Building codes do require that doors be lockable, however.
So requiring that NICs have the capabilty might be ok.

>    clear violation of layering that should be avoided

The DDP Layer MUST NOT place data for any DDP message
unless:

a) the LLP messages containing the header information have
been fully validated. I maintain that LLP messages can be
validated out of order, even theoretically, because
retransmission is not allowed to alter the correct contents
of the message.

b) At least the DDP header has been validated. Without your
seperate header, this requires that the entire DDP segment
has been validated. Which would mean any LLP message
containing part of the DDP segment must have been validated.
For example, it would be incorrect to process a DDP segment
that had been completed by an IP fragment that contained the
rest of the DDP segment, but did not complete the TCP
segment. Admittedly this is almost a malicious middlebox
scenario.

Buffers are dynamically divisible by the ULP in ways that
cannot be pre-described to the hardware.

For example: a ULP data sink could define a large circular
buffer, and enable the data source ULP to place within it.
Under the ULP rules, untagged messages from the data source
report what section has been updated, and untagged messages
from the data sink restore the writability of the buffer.

That is a simple, reliable, ULP-specific protocol for
unambiguously assigning ownership of each byte within a
dynamically divisible buffer. More importantly it is just
one example of such schemes that are more complex than a
"enable buffer" / "buffer done" life cycle.

There is simply no way for the hardware to support every
valid buffer strategies that ULPs would wish to employ.

Caitlin Bestler
http://asomi.com/CaitlinBestler/
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: Bill Sommerfeld <[email-masked]>
Date: Wed, 20 Nov 2002 13:49:27 -0500

[my primary reason for being involved with this working group is to
attempt to make sure that any RDDP technique does not introduce new
system-level vulnerabilities.  as far as I can tell, the simplest and
most robust way to do this is with clear buffer ownership rules..]

I'm in violent agreement with all of the above.

This is simpler to define but seems to be a hair too strong -- simply
locking out writes to buffers which have been released to apps seems
to be sufficient.  What attack were you thinking of?

					- Bill

---

From: Caitlin Bestler <[email-masked]>
Date: Thu, 21 Nov 2002 13:10:44 -0600

On 11/20/02, Bill Sommerfeld wrote:

>   a buffer with the DDP layer also passes ownership of 
>   that buffer.  The application is not allowed to peek or 
>   poke any buffer that the DDP layer owns.
>
>   ownership of the containing buffer back to the 
>   application layer. At this point DDP may not write any 
>   more data in that buffer.
>
>This is simpler to define but seems to be a hair too
>strong -- simply locking out writes to buffers which have
>been released to apps seems to be sufficient.  What attack
>were you thinking of?
>

Transferring ownership of advertised buffers is already part
of the semantics of DDP. The contents of advertised buffers
are non-deterministic to the Data sink ULP until there is a
completion delivery. If you want to call that "ownership" by
the ULP, that's fine, as long as you recognize that the ULP
can revoke that 'ownership' at any time. And further, the
ULP's 'ownership' of that buffer can be revoked by the OS
at any time.

The questions are:

1) how is DDP 'ownership' enforced?

2) Does DDP have to be aware of individual advertised
buffers within a single STag defined region?

Question 1 - Enforcement:

Removing access privileges to advertised buffers would
require extreme integration with the OS. It would require
user-mode calls to temporarily suppress access privileges at
a byte granularity level. If you have man pages for any such
routines please share them. We can then test to see if they
are implemented with sufficient efficiency to be usable on a
per transaction basis.

Such motherboards do not exist. They won't either, because
such checking would be signifigantly more difficult than the
checking an RNIC has to perform.

An RNIC has to check the bounds of a single supplied STag
once for each tagged segment. A MMU enforcing STags would
have to check *each* memory access against *all* STags.
I believe it is easy to prove that such checking *cannot*
be accomplished without a parallel memory system that is
*superior* to the base memory system. The problem is that it
would have to be large enough that it would have to be
stored in external memory...

The current drafts set realistic goals for hardware, and
co-exist with host OSs as they are. It allows a powerful
new feature to be added to existing systems without
requiring that they be rebuilt from scrach.

The bottom line on DDP security:

    DDP does not enable a process to have any rights that 
    the OS did not give it.
    
    DDP has no rights that the ULP did not give it, either
    by creating an STag or preposting a read for an untagged
    message.

The process can fully protect itself from its remote peers,
by creating narrow STags and promptly invalidating them, The
process is also presumably protected from its local peers,
by the host OS.

Question 2 - Single STag - many buffers

The use of a single persistent STag encompassing multiple
advertised buffers is a legitimate programming model. It is
not merely a cheap trick to minimize STag creation/deletion
overhead.

An important feature of an advertised buffer is that it is
divisible. This allows the Data Source to use less than is
advertised and only return the portion used to the Data
Sink.

I will give *one* pragmatic example, but there are countless
application specific uses for this capability.

The Data Sink ULP is a streaming media client. It wishes to
receive a variable rate multimedia presentation that is
encoded with multiple different media streams (ULP sense of
stream here: video stream, audio stream, text stream...).

It sets up a circular buffer for each media stream, and
creates an STag for each of them (we're assuming that STag
creation is so low overhead that it prefers to create them
seperately).

It then connects with the server, and uses untagged messages
to establish a delivery session. It provides the STags for
each media stream, and advertises the entire buffer in each
as being availble.

The Data Source ULP begins doing tagged placements into the 
destination buffers. Periodically it sends an untagged
message confirming delivery to a certain point in the
presentation. This untagged message would list the committed
portions of each buffer, and presumably a timing mark. In
doing so the Data Source ULP is returning ownership of those
portions of the buffer to the Data Sink ULP.

The Data Sink ULP is always waiting for untagged messages,
and notes the progress reported with each. Meanwhile the
playback thread is consuming the media streams in a live
playback to the end consumer. Whenever it begins a new
packet it checks to see if that portion has been committed
yet. Over a reliable connection it would stall until the
material was available. Once unreliable service is defined
it would have the option of doing error concealment instead.

Periodically, the Data Sink ULP sends an untagged message
that announces what it has consumed for each thread, thereby
re-enabling that portion of the buffer for further delivery.

Some important ULP specific characteristics here: the only
signifigant boundaries are entire ULP packets (the playback
thread checks at the begging of ULP packets only).

Per transfer STags would not be feasible, because the Data
Sink ULP has no idea how big each packet would be. Using per
transfer STags, or untagged transfer, each receive buffer
would have to be the maximum for each time slice. With
variable encoding this could amount to a signfigant amount
of extra memory. Further,the playback engine would have to
be more complex to deal with jumping over the wasted memory.

And oh yes, the Data Source ULP has the content stored on
multiple drives. Therefore the order of delivery of the
content within any time period is unpredictable. It only
guarantees that all material for the time slice will be
delivered in time.

Challenge: can you specify an efficient enforcement scheme
that is compatible with this type of application? How about
with the other applications that I didn't list or didn't
know about?

This is just one example of where a ULP can reliably have
deterministic knowledge that a portion of an STag enabled
region is in fact not subject to remote update at this time.

In this example it is taking the "risk" of using a single
STag for a circular buffer, knowing full well that if its
partner is malicious it could trash the very material it had
previously delivered.

I think we need to trust the ULPs to know the difference
between a streaming media playback, and transactional
records that must be guaranteed to be non-refutable. It is
the ULPs job to know when it is important to invalidate
STags, not DDPs.

Caitlin Bestler