Subject: [rddp] When the Host is a middlebox

From: Caitlin Bestler <[email-masked]>
Date: Sat, 23 Nov 2002 10:28:23 -0600

One of the requirements for DDP is that it be transparent to
middleboxes. That is, DDP message segments appears to be
normal TCP or SCTP traffic. No special handling may be
required.

There is at least one "middlebox" that this is problematic
for: the existing IP stack on the host itself.

The existence of a host IP stack independent of the DDP
stack is unique to RDMA over IP, and I believe inevitable.
It also creates a few problems that at the minimum are big
warnings for the implementors guide. While the host IP stack
may not have to be *altered*, it will at the minimum have to
be *configured* in certain ways to maintain interoperabilty.

I believe the scenarios I will outline justify the following
two requirements:

-- The DDP implementation MUST accept incoming IP packets
from the host. It MUST NOT assume they will arrive only upon
its external port.

    This is actually implied by stating that it runs over
    the LLP. But it is a non-obvious requirement that should
    be highlighted.
    
-- The DDP implementation SHOULD inform any host based IP
routing of which connections are in DDP mode.

    This is a major efficiency issue.

Why the problem is unique to RDMA over IP:

Other RDMA networks have been based upon the ULP
communicating directly with the RNIC. The Host OS is
involved only to validate that the ULP has authority to talk
with the RNIC driver, to support RNIC callback/interrupts to
the ULP, and to determine the mapping from virtual to
physical memory.

But both Sends and receives operations are posted directly
to specific RDMA devices or endpoints/work queues they
control. In contrast, typical IP interfacing is with the
host IP stack, which selects the L2 driver to deliver the IP
packet to. And typically the IP stack receives IP packets
from any L2 device, subject to its own firewall inspection
rules.

One even more important difference: IB hosts are not IB
Routers. IP hosts are frequently IP routers, in fact many
view this as the default configuration for IP hosts.

Why a seperate host IP stack is inevitable:

The host IP stack already exists. It already supports LLP
connections on one or more L2 devices. There are already
applications relying upon those services.

Allowing a new application that wants to do something called
"RDMA" to install a special driver for a special NIC is one
thing. Allowing this new application to replace the entire
IP stack for every interface is quite another. To put it
mildly, it is not likely to be greeted enthusiastically by
system or network administrators.

So the existing IP stack will remain in place. In fact the
new RNIC is likely to also play conventional NIC. It will
exchange Ethernet Frames with the existing host-based IP
stack whenever it has not pre-empted an LLP connection.

Why this is a problem - consider the following scenario:

-   The host has an existing NIC X on subnet J.
-   The host has an RNIC Y on subnet K.
-   The host is a valid route to subnet K from subnet J.

When an IP packet destined for RNIC Y's IP address arrives,
how should the host route it?

To itself? This is probably what it would do by default. But
that would be incorrect for a DDP connection.

To RNIC Y? If this is a non-DDP connection then the packet
will be crossing to the DDP and back again. This is hardly
an efficient solution.

The local routing decision either could be based on
dynamically updated port-specific rules. Most IP hosts that
route packets have port-specific firewall capabilities, and
these rules can typically be dynamically adjusted. Under
this scenario, no changes to the host IP stack are actually
required. But the DDP stack must be enabled to configure
port-specific routing so as to ensure that its packets will
be routed to it.

Note that Jim Williams' earlier suggestion of providing the
RNIC with its own IP address is an even better solution
here. It SHOULD be used when extra IP addresses are not a
problem (i.e. with private IPv4 or IPv6). Use of an IP
address that is specific to the RNIC would eliminate any
need for dynamic firewall rule twiddling.

Another possibility would be to allow the host IP stack to
know for itself which connections are in DDP mode. This is
more efficient, but does involve a change to *the* host IP
stack. Each project can determine for itself how many
quarters must be added to their testing schedule if they
require changes to *the* host IP stack.

Caitlin Bestler
http://asomi.com/CaitlinBestler/
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp