Subject: [rddp] RE: SCTP Multi-homing issues

From: "Williams, Jim" <[email-masked]>
Date: Tue, 29 Oct 2002 12:38:21 -0500

> -----Original Message-----
> From: Caitlin Bestler [mailto:[email-masked]]
> Sent: Tuesday, October 29, 2002 11:08 AM
> To: [email-masked]
> Subject: [rddp] SCTP Multi-homing issues
> 
> 
> The multi-homing capabilities of SCTP raises some questions
> for DDP and the accessibility of host memory across multiple
> interfaces.
> 
> An SCTP endpoint on a multi-homed host is a collection of
> eligible IP addresses. Presumably these are on different
> physical interfaces, doing otherwise just slows down the
> failover process that is supposed to provide fault tolerance
> for network failures.
> 
> This raises a few possibilities:
> 
> -- An RDMA/DDP SCTP endpoint may only select IP addresses
>    that are managed by the same RNIC. This would represent
>    a restriction on RDMA/DDP SCTP associations that would
>    not apply to other SCTP associations.
>    
> -- All STags enabled for any stream in the association must
>     be enabled for ALL RNICs that are part of the SCTP
>     endpoint. This would require that SCTP RNICs accept
>     host processor assigned STags.

A third alternative is that the RNIC is a separate
endpoint (SCTP or TCP), with its own IP address
distinct from the host's IP addresses.  Therefore
multiple RNICs are separate endpoints and not
eligible to part of the same SCTP association.

If you take the other approach where the RNIC shares
an IP address with the host network stack, you are
left with a list of difficult problems.
Examples of the types of problems that occur include:

    How does the RNIC derive the ID field in the IP header
    so it doesn't collide with host generated IDs?
    
    What happens if the host is configured as a router
    and a packet destined for a connection accelerated
    by a given RNIC arrives on a different NIC?
   
    How do you manage the connections (SNMP etc.) to give
    a consistent view of statistics when some connections
    are offloaded by the RNIC and others run on the host
    stack?
    
    What happens if the host attempts to use link
    aggregation across two different NICs?

    The SCTP issues you raise above.

There may be practical solutions to all these problems,
but the number and complexity of such problems is troubling.
I would be interested in hearing how others perceive
these issues.
    
These problems go away if the RNIC is a separate endpoint
from the host.  However that means fail over
must be done at the application level,
because fail over at the RDMA, DDP, or transport
layer does not work.

> If my understanding of the feature is correct, an SCTP
> endpoint must be ready to receive packets on any of the
> eligible interfaces at any time. This is a sharp contrast to
> automatic path migration under InfiniBand. It also mandates
> full real-time duplication of all STag related resources on
> every interface that is part of an SCTP endpoint. This is
> clearly undesirable if the alternate interface was intended
> solely as a fallback (i.e., a primary interfacewould carry
> the entire traffic, as opposed to having n interfaces load
> balance the traffic and willingly reduce to n-1 interfaces
> with a reduced capacity).
>  
> In my opinion, many to most RDMA ULPs would prefer to handle
> path migration on their own. This is especially true of
> those that are using RDMA to achieve very low predictable
> latencies. For those ULPs, restricting an SCTP endpoint to
> the scope where the hardware can pool STag resources "for
> free" would be the optimal choice.
> 
> For others, fully conforming to standard SCTP behavior with
> its fault tolerance and congestion management would be
> preferred.
> 
> Does anyone else have any thoughts on this tradeoff?
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: Caitlin Bestler <[email-masked]>
Date: Tue, 29 Oct 2002 15:26:42 -0600

>eligible to part of the same SCTP association.

In the context of the question raised above, that's just a
subset of the first case: the endpoint is limited as to
which IP addresses it may use.

>    so it doesn't collide with host generated IDs?

    One approach is that the RNIC does not truly own the
    connection/association, it merely handles most of the
    packets. Hence critical setup items, such as the choice
    of IP addresses or the SCTP verification tag are under
    control of the host processor.
    
    Having the RNIC take full responsibility for the
    connection/association is nice, but hardly necessary
    to ensure performance. Many aspects are only handled
    once per connection.
    
>    by a given RNIC arrives on a different NIC?

    The ULP must place a stream/connection in RDMA/DDP mode. 
    It  can only do so if their is an association/connection
    with this host. IP packets that are for other machines 
    are not targeting an association/connection on this 
    host, and hence cannot be subject to RDMA/DDP 
    processing.
    
>    are offloaded by the RNIC and others run on the host
    A necessary local interface, but not necessaarily one
    that requires standardization.
    
    Giving the RNIC its own IP address, and own MIB, is
    certainly one excellent solution to this problem.
       
>    aggregation across two different NICs?
    Link  aggregation is not defined for TCP. The host
    would have to have an internal IP address and 'route'
    packets via more than one external interface.
    
    SCTP specifies that 'By default, an endpoint SHOULD 
    always transmit to the primary path, unless the SCTP 
    user explicitly specifies the destination transport
    address (and possibly source transport address) to use.'
    
    Which means that link aggregation could easily happen
    at *either* the ULP or SCTP layers.
    
>    The SCTP issues you raise above.

Implementing DDP/multi-home SCTP is actually not that
frightening, as long as all of the issues are defined.
Each RNIC would be responsible for placement, but completion
processign (at both the SCTP and RDMA/DDP layers) would have
to be a common process.

There are many ways to provide that common process, and
ensure that it is efficient enough. The real issue is the
duplication of STag resources and the fact that for an STag
to be meaningful to the ULP it must be valid on all RNICs
that might be used for an association.

>from the host.  However that means fail over must be done
>at the application level, because fail over at the RDMA,
>DDP, or transport layer does not work.
>
Failover of the primary path to an alternate path within an
SCTP association would not create any problem that I can
think of.

Caitlin Bestler
http://asomi.com/CaitlinBestler/
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp