Subject: Re: [cicm] Channel I/O

From: "Lanz, Dan" <[email-masked]>
Date: Thu, 17 Jun 2010 12:30:36 -0400

Another related question is whether the I/O functions have been defined for maximum performance.  Consider CICM::Encrypt::Stream::encrypt(), which accepts data for encryption, and CICM::Decrypt::Stream::decrypt(), which receives decrypted data.  Are the IDL definitions such that a high performance implementation would be possible?

Dan

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Novikov, Lev
Sent: Wednesday, June 16, 2010 3:32 PM
To: CICM Discussion List
Subject: [cicm] Channel I/O

Terminology:
- CHANNEL: the most general abstraction for a connection between a module and software.
- CONTROLLER: a channel that has management capabilities.
- STREAM: a channel that has data-passing (sending, receiving or both) capabilities.
- CONDUIT: the channel that has both management and data-passing capabilities 
	(the logical union of a controller and stream).

(See: Section 2.3.3, http://tools.ietf.org/html/draft-lanz-cicm-00#section-2.3.3)

I/O, therefore, is done primarily by a stream or a conduit and consists of two modes:
blocking and non-blocking. BLOCKING refers to a mode where the method does not 
return until the buffer is full. For example, Decrypt::Stream::decrypt()
does not return until the buffer is full (or an error occurs). Similarly,
Encrypt::Stream::encrypt() does not return until the operation completes (or
an error occurs).

NON-BLOCKING refers to a mode where a buffer is registered, and periodically polled
to determine its status. Decrypt::Stream::decrypt_non_blocking() performs the
registration/filling while Decrypt::Stream::decrypt_poll() returns the status.
Encrypt::Stream::encrypt_non_blocking() and Encrypt::Stream_encrypt_poll() 
perform in a similar fashion.

* Are these modes sufficient; are others necessary?
* Are there any implementation issues with these modes?

Lev
_______________________________________________
cicm mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/cicm

---

From: "Dr. Joe Mitola" <[email-masked]>
Date: Thu, 17 Jun 2010 15:07:58 -0400

Just my two cents, but:
Realization of IO functions depends on the sophistication of the 
implementer and perhaps on the insights and proclivities of the 
acquisition and regulatory bodies. 
IDL definitions can be interpreted as "how" meaning that given IDL, one 
must use CORBA to realize the interface, and that can be incredibly 
inefficient.
Alternatively, IDL definitions can be interpreted as "what" so that if 
two applications exist in the same processor or in two different 
processors that share memory, one may "compile out" the IDL and the 
CORBA by providing a direct DMA/interrupt based path between the two 
processors, app to app, with microsecond class latency versus CORBAs 
order of magnitude or worse overhead.
The speeds of the hardware and the IDL are independent as well, so 
unless the IDL attempts to constrain the implementations aggressively, 
there is no reason that IDL precludes high performance.
However, if institutions (like a defence department) choose to interpret 
IDL as how versus what, then this unnecessarily constrains realizations.
So unless we do something aggressive with IDL to constrain the 
implementations, there's nothing inherent in CORBA or IDL that forces 
inefficiency.
At least this is the experience of the radio engineering community with 
the OMG SRA (software radio architecture).
joe

Dr. Joseph Mitola III, Fellow of the IEEE
Distinguished Professor
Charles V. Schaefer, Jr.  School of Engineering and Science
School of Systems and Enterprises
Vice President for the Research Enterprise
Stevens Institute of Technology
Castle Point on Hudson, Hoboken, NJ, USA
Cell: 703-314-5709

---

From: "Novikov, Lev" <[email-masked]>
Date: Thu, 17 Jun 2010 15:32:53 -0400

> implementations, there's nothing inherent in CORBA or IDL that forces 

That's good news and it matches our original intent in using IDL; we 
explicitly state that CORBA is not being mandated, but if CORBA is 
going to be used, the IDL is normative (as in "how"). Otherwise, the 
IDL simply specifies the "what."

Lev

---

From: "Dr. Joe Mitola" <[email-masked]>
Date: Thu, 17 Jun 2010 17:02:40 -0400

The 'what' can be normative, Lev.  Better to have the what part 
normative vs the how part which can be advisory

---

From: "Novikov, Lev" <[email-masked]>
Date: Fri, 18 Jun 2010 16:02:28 -0400

> normative vs the how part which can be advisory

Agreed; only the IDL definitions (what) are normative.
The specification (Section 1.1) says:
   The use of IDL in CICM is not intended to either prescribe or
   preclude a particular communications protocol such as General Inter-
   ORB Protocol (GIOP) between programs in different address spaces or
   on different devices.
See: http://tools.ietf.org/html/draft-lanz-cicm-00#section-1.1

The "how" is almost considered out-of-scope, but we wanted to make sure
that the current definitions don't obstruct the use of high performance
implementations (which it shouldn't as you've clarified).

Thanks,
Lev

---

From: "Linn, Charles" <[email-masked]>
Date: Tue, 22 Jun 2010 13:56:34 -0400

An area I can provide some insight, yet not a solution comes from the
JTRS realm.  Without going into specifics, while a commonly used
interface passes data to the API as an octet sequence for encryption /
decryption, there are also some implementations out there that use a
different zero-copy method (this also plays in with the "high assurance"
side of things, but can't discuss why, sorry).  Security considerations
put aside, zero copy is not only faster, but (when it can be used)
allows for nagleing and other network packet techniques, where the
packet is constructed "oversize", and grown in place as headers are
added, etc.

With this technique, rather than an octet sequence, some sort of opaque
index is passed. This index is later dereferenced in the implementation
to access the data (usually in a shared memory area, etc).

-Chuck Linn

---

From: "[email-masked]" <[email-masked]>
Date: Wed, 23 Jun 2010 04:35:43 -0400

Hi Chuck,

It was the design intent of the CICM APIs to support zero-copy. The
realization of this intent is probably far from obvious in looking
at the APIs.  Lev asked me to chime in to provide some background
so that you and others can evaluate the success of CICM at achieving
this goal.

As you are probably aware zero-copy APIs from transports may impose
a variety of different and sometimes complex constraints on how the
applications and systems implementing and using these APIs are
architected. These constraints frequently result in requirements like
the using application must arrange for data to arrive or be
generated in a type of memory (eg DMA-able) or location of memory
that anticipates the destination of a subsequent transmission.
There are frequently strict alignment constraints around page, cache,
or other boundaries. In addition, the reclamation and the future
reuse of any memory used to transmit data is usually deferred until
some future point (typically the completion of when some other
component completes the transmission). Every zero-copy system I
have encountered seems to provide innovative and different mechanisms
for managing and indicating memory reuse or reclamation.

As a result it is incredibly difficult for any API to accommodate
all of the vast array of constraints that are imposed by various
zero-copy communication technologies such as RDMA, page movement,
clustered buffering, etc.  A classic example is MPI, an API designed
specifically for zero-copy systems. Yet the vast majority of MPI
implementations make at least one copy of the data.  The MPI model
can accommodate a certain class zero-copy systems.  But rarely do these
zero-copy systems fit within the MPI model faithfully.

The idea in CICM was to accommodate zero-copy through some of the
flexible semantics of some of the OMG IDL mappings of the sequence type.
For example, in C++ a sequence<octet> results in a holder class that
manages a buffer pointer.  This holder class can accommodate a wide
variety of memory types, sources of memory, and memory reclamation
semantics. Similar semantics exist for other IDL language mappings.

WRT to your suggested semantic of passing a memory handle instead
of a pointer (as with sequence<octet>)... The sequence<octet>
implementation provides much of the semantics for how the application
writes to or reads from the referenced memory. The actual memory
under a sequence<octet> does not have to be a pointer but could
be a memory handle. So I think sequence<octet> might be able to
allow for your implementation semantics without additional overhead.

But one challenge that may exist in this situation is that your memory
model may require additional semantics on the application using
the sequence<octet> with your system.  It would be good for you
to communicate to Dan and Lev if there are additional semantics
that your memory design would require. If these semantics require
use of additional API extensions then Dan and Lev can either (i) help
determine if the existing CICM APIs can accommodate the semantics,
(ii) whether they want to add/modify APIs to accommodate the semantics,
or (iii) suggest ways you can extend the CICM APIs (since the framework
is extensible) to accommodate the semantics.

Hope this is useful.

Best regards,

Bill

---

From: "Linn, Charles" <[email-masked]>
Date: Mon, 28 Jun 2010 09:46:54 -0400

Hi Bill,

Good insight, and presents some variations I had not seen before.  I
agree that this is a hard problem, and it is difficult, and probably
impossible to come up with a single API that "actually specifies
something" that can accommodate most reasonable implementations.

One area you can perhaps explain a bit more.  From a waveform
portability standpoint, an API must not only relate a signature, but
also have sufficient semantic and behavioral detail so as to not cause
undue porting between differing interpretations or implementations of
the API.  There are two ways I read into your response - can you
clarify?

In the first view, the octet sequence stays a "standard" sequence, but
the payload of what is placed inside changes to, say, some sort of an
index, etc.  In this case, unless I am missing something we didn't have
to change the API, true, but the semantics to use it HAVE changed, such
that the waveform would need to be coded to the implementation - pass in
the live data, or pass some handle to the data.

The second interpretation I had not thought of, still evaluating the
possibilities.  In this case, the waveform places the data into an octet
sequence, but.. the ORB implementation of the sequence places the data
is some sort of shared memory pool, and then, invisibly to the clients,
marshals it into a handle, index, etc.  At the receiving end, the
reverse is performs and the data is reconstituted.  I am not sure if
this would serve as a good solution to "zero copy", but it may be
possible for such solutions to work from a security standpoint for
memory wiping, etc.

Interesting discussion either way, thanks for the thoughts.

-Chuck

---

From: "[email-masked]" <[email-masked]>
Date: Mon, 28 Jun 2010 11:33:33 -0400

> the API.  There are two ways I read into your response - can you

I agree that clear semantics and behavioral detail are required for
good waveform portability.  But I view portability as shades of grey.

Rather than try to describe my views on portability with prose
descriptions I'll try to define an example taxonomy of portability
that might help you and others understand my responses to your
questions.

I think about portability in various levels of success (I'm
not sure the ordering is perfect):

1. Waveform binary code compatibility without additional copies.

   This is where a set of object code logic will operate
   correctly with just a relink or reload and will not require
   additional copies when used in the new system.

2. Waveform binary code compatibility with additional copies.

   This is where a set of object code logic will operate
   correctly with just a relink or reload but will result
   in additional data copies when used in the new system. The
   intent here is that the waveform object code would not be
   able to tell that the CICM support system is forced to make
   copies on its behalf since the application is not following
   the semantics preferable to the new system's mode of operation.

3. Waveform source code compatibility without additional copies.

   This is where a set of source code logic will operate
   correctly with a recompile and will not require
   additional copies when used in the new system.

4. Waveform source code compatibility with additional copies.

   This is where a set of source code logic will operate
   correctly with a recompile and but will result
   in additional data copies when used in the new system. The
   intent here is that the waveform source code would not be
   able to tell that the CICM support system is forced to make
   copies on its behalf since the application is not following
   the semantics preferable to the new system's mode of operation.

5. Waveform source code changes required to avoid additional copies.

   This is where a set of source code logic needs modification
   in order to avoid additional copies when used in the new system.

6. Waveform source code changes required to adapt to feature/function
   differences in CICM implementations.

   This is where a set of source code logic will not operate
   correctly without adaptation to a different set of supporting
   logic in the new system's CICM implementation.

7. Rewrite your waveform

> the live data, or pass some handle to the data.

I believe that in many cases CICM implementations can achieve portability
levels (1) and fall back to (2) with most waveforms. This would require
the CICM implementation to understand the kind of memory and appropriate
semantics of the kind of memory required for the system. Then the CICM
implementation would magically know when it can avoid copies and when
it would need to copy.

> possible for such solutions to work from a security standpoint for

This would be a fine example of an implementation that a CICM implementation
(I'm including the ORB in your example as part of the CICM implementation)
would not achieve total zero copy but would impose little additional
memory constraints on the waveform.

> Interesting discussion either way, thanks for the thoughts.

Glad to help. It would have been hard to write all this interpretation
in the CICM spec without leading folks to believe that the thoughts behind
these mechanisms were constraints on implementations.  Instead they are
meant to enable. Initial CICM implementors can best guide and correct
any flaws in this enablement.

Best regards,

Bill