Subject: Re: [Ipoverib] A Couple of IPoIB Questions

From: Vivek Kashyap <[email-masked]>
Date: Tue, 16 Nov 2004 23:38:29 -0800

Hi,

I have a couple of questions relative to IPoIB:

1. draft-ietf-ipoib-ip-over-infiniband-07.txt states:
"Every IPoIB interface MUST "FullMember" join the IB multicast group
defined by the broadcast-GID."

Isn't the broadcast group for IPv4 ? When the IPoIB interface is IPv6 only,
does this group still need be joined ?
If not, where do the parameters for any IPv6 groups come from ? I am
presuming that this group needs to be joined in
      the IPv6 only case. I just want to be sure.

      <VK> Yes, the broadcast-GID is at the InfiniBand layer and MUST be
      joined whether you are running at v4 or v6 layer. <VK>

      2. ALso, what is the latest status of the Vivek's connected mode
      draft ? Will it be moving forward ?

      <VK> I'll be submitting it as draft-ietf-ipoib-connected-mode-00.txt
      by the end of the month. There were some interesting suggestions that
      were made during the IETF WG meeting. Two of the suggestions of
      consequence are given below. The others we can discuss when the
      minutes are published (they include some additional requests on
      clarification on the transmission draft too).

      a. The current draft makes the various modes mutually exclusive i.e.
      RC, UC and UD are not allowed simultaneously in the same IP subnet.
      The thought is that it is a link characteristic and hence different
      per connection mode. It was suggested that one be allowed to mix up
      RC/UC. This goes back to the original suggestion in the first draft
      which was:

      IPoIB-UD must always be supported. Additionally, the interface can
      also support either both of RC and UC, or one of them. Or neither of
      them.

UD MUST always be supported.

<VK> That is and has always been the requirement right from the first
draft. <VK>

 I personally don't care whether one does RC or UC but I don't think both
are required as a MAY option.  The advantage of RC is the send credit
algorithm.  The advantage of UC is the lack of ACK packets.  ACK is noise
in the fabric while send credits provide a simple method to maintain
bandwidth / injection control on a per flow basis.

I see no problems with supporting both UD and *C on the same subnet; it is
rather foolish to attempt to mandate these be on separate subnets.b

   <VK> As per the connected-mode draft the UD mechanism is *always*
   required; address resolutoin depends on it.

    The only point of discussion is whether all nodes must support the same
   link characteristics in the subnet i.e. all are RC (and UD), or all or
   UC (and UD), or all are UD only. The alternative is to allow all the
   nodes to be mixed up with some nodes being RC/UD, others UC/UD and a
   third set UD only  and yet others probably supporting all. within the
   same IP subnet. [Can the same serviceID be used by both RC and UC ?]

   The third alternative is to associating UD only or UD +  one of RC or UC
   on the same interface. In such a case if mismatched/unsupported
   connected modes are supported by two nodes then the fall back to UD.
   This option is not too different from UD QP + RC or UC mechanism.

   <VK>

      b. Another suggestion was to allow multiple connected mode links
      (i.e. at IB UC/RC level) between peers.

      One thought can be 'yes, but user beware': The IB connections are
      made using the service ID that is derived from the QPN as described
      in the draft. If a second attempt succeeds then there are two links.
      It is up to the implementation to either allow or disallow multiple
      links.

Again, this has been suggested in the past (though most who were involved
in the original discussions years gone by are likely gone since much of
this discussion occurred before the IETF workgroup was established).

<VK> I'm one of the vestiges of those early times along with you and a few
others...so we have hope :). <VK>

 There is obvious benefit to supporting multiple RC per endnode pair.  I do
not see any technical reason to oppose nor any issue from an
interoperability perspective.  There is no reason for a "user beware".

<VK> It is not opposed. The 'user beware'  is only underscoring that the
the peer interface might not support multiple links- it might enforce a
limited number of connections (maybe only one) between a pair of GIDs.
Similarly, an implementation not wanting to support multiple links MUST
take steps to deny multiple requests.

<VK>

 The work is rather straight to do and implement and the benefit to
customers, is again, rather obvious when one considers what the IB fabric
offers and how connections can be enable flows through multipath as well as
transparent fail-over, flow scheduling, mapping of DiffServ to different
arbitration / paths, etc.

<VK>  In addition Large MTU and APM are two of the main reasons why I've
been proposing IPoIB-connected mode for so long. In terms of IPoIB itself,
except for the Large MTU, the parameters are hidden from it.<VK>

Mike
_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib

---

From: Michael Krause <[email-masked]>
Date: Wed, 17 Nov 2004 16:46:52 -0800

At 11:38 PM 11/16/2004, Vivek Kashyap wrote:

>Hi,  I have a couple of questions relative to IPoIB:  1. 
>draft-ietf-ipoib-ip-over-infiniband-07.txt states: "Every IPoIB interface 
>MUST "FullMember" join the IB multicast group defined by the 
>broadcast-GID."  Isn't the broadcast group for IPv4 ? When the IPoIB 
>interface is IPv6 only, does this group still need be joined ?  If not, 
>where do the parameters for any IPv6 groups come from ? I am presuming 
>that this group needs to be joined in  the IPv6 only case. I just want to 
>be sure.
><VK> Yes, the broadcast-GID is at the InfiniBand layer and MUST be joined 
>whether you are running at v4 or v6 layer. <VK>  2. ALso, what is the 
>latest status of the Vivek's connected mode draft ? Will it be moving 
>forward ?  <VK> I'll be submitting it as 
>draft-ietf-ipoib-connected-mode-00.txt by the end of the month. There were 
>some interesting suggestions that were made during the IETF WG meeting. 
>Two of the suggestions of consequence are given below. The others we can 
>discuss when the minutes are published (they include some additional 
>requests on clarification on the transmission draft too).  a. The current 
>draft makes the various modes mutually exclusive i.e. RC, UC and UD are 
>not allowed simultaneously in the same IP subnet. The thought is that it 
>is a link characteristic and hence different per connection mode. It was 
>suggested that one be allowed to mix up RC/UC. This goes back to the 
>original suggestion in the first draft which was:  IPoIB-UD must always be 
>supported. Additionally, the interface can also support either both of RC 
>and UC, or one of them. Or neither of them.
>
>algorithm. The advantage of UC is the lack of ACK packets. ACK is noise in 
>the fabric while send credits provide a simple method to maintain 
>link characteristics in the subnet i.e. all are RC (and UD), or all or UC 
>(and UD), or all are UD only.

Obviously I would oppose such a solution as it creates artificial 
constraints with little benefit.

>The alternative is to allow all the nodes to be mixed up with some nodes 
>being RC/UD, others UC/UD and a third set UD only and yet others probably 
>supporting all. within the same IP subnet. [Can the same serviceID be used 
>by both RC and UC ?]
>
>The third alternative is to associating UD only or UD + one of RC or UC on 
>the same interface. In such a case if mismatched/unsupported connected 
>modes are supported by two nodes then the fall back to UD. This option is 
>not too different from UD QP + RC or UC mechanism.

KISS:

- UD universal
- *C opportunistic
         - Local management issue to control what is sent on the *C 
interface.  No need to specify
         - Advertise whether one or more ports are supported by UD or *C
         - Advertise whether one or more QP are supported by UD or *C
         - Let local management determine policy for what services are 
mapped where - no need to specify

This is both an interoperable approach and simple to implement.  There may 
be some desire to add a policy interface to state preference for specific 
types of traffic over a given QP.  I would not oppose this but would view 
this as a separate draft once the basics are worked out.

><VK>
>b. Another suggestion was to allow multiple connected mode links (i.e. at 
>IB UC/RC level) between peers.  One thought can be 'yes, but user beware': 
>The IB connections are made using the service ID that is derived from the 
>QPN as described in the draft. If a second attempt succeeds then there are 
>two links. It is up to the implementation to either allow or disallow 
>multiple links.
>

*C requires CM to operate thus it is a local issue whether additional CM 
operations are accepted or not.  A given requester node may issue N and a 
given responder may state 0-N as an implementation may limit the number of 
*C available for IP traffic.

>offers and how connections can be enable flows through multipath as well 
>as transparent fail-over, flow scheduling, mapping of DiffServ to 
>different arbitration / paths, etc.
>

Mike

---

From: Vivek Kashyap <[email-masked]>
Date: Wed, 17 Nov 2004 22:46:49 -0800 (Pacific Standard Time)

Mike the format is really off in the last mail from you - making it difficult
to follow.

Other than that let us discuss in the context of the draft. The draft is
built upon the following:

1. IPoIB-RC and IPoIB-UC are optional.
2. IPoIB connected mode depends on a UD QP for address resolution and multicast.

As far as I know, there has been an agreement since the earliest connected mode
draft I posted.

I'd like the WG to give input on the following issues:

3. Where does the UD QP come from?  Choose one of:

a. It is a UD QP that is associated with the interface at startup.

b. It is a UD QP that is shared with IPoIB-UD.

3a is more generic. It can be considered to include the case 3b.  The original
proposal was limited to 3b.

4. Link characteristics

The broadcast domain for IPoIB-RC/UC is determined exactly as the
IPoIB-UD case i.e. through the broadcast-GID. A UD as per 3 is used in this
step.

Do all interfaces in the IPoIB-conneced mode(CM) have the same link
characteristics? i.e.

a. all are either IPoIB-RC or IPoIB-UC.

	-- There is also a UD QP associated. The UD QP will be either 3a or 3b
	   based on WG concensus.

	-- All unicast transmission is on the IPoIB mode i.e. RC or UC.

b. all are IPoIB-UD. Additionally they can be one of IPoIB-RC or IPoIB-UC
or both.

	-- The presence of the flags indicate the type of communication possible.
	-- The decision of communicating using a specific mode is determined by
	   the supported modes and the local policy. Note that incompatible
	   policies imply that the fallback is communication over UD.
	-- fallback mode of communication is UD

4b adds a lot of flexibility at the expense of a simple decision. 4a. by
contrast is straightforward.

5. MTU negotiation

	In the private data field of the CM message the desired MTU is
	included.

	It was suggested during the IPoIB meeting at IETF that it need not be
	symmetric. That is a good idea. Thus each peer declares the max MTU it
	prefers

	REQ: <my desired MTU>
	REP: <my desired MTU>
	RTU:

6. Multiple connections for the same IP address

	Local decision. Note that the peer might choose to not honour multiple
	connections.

Vivek

> >except for the Large MTU, the parameters are hidden from it.<VK>

__

Vivek Kashyap
Linux Technology Center, IBM

---

From: Roland Dreier <[email-masked]>
Date: Thu, 18 Nov 2004 06:56:54 -0800

> Mike the format is really off in the last mail from you -
    > making it difficult to follow.

Vivek, I think that if you used standard quoting in your replies
instead of your own "<VK>" format, it would be much easier to follow
email threads involving your replies.

Thanks,
  Roland

---

From: Michael Krause <[email-masked]>
Date: Thu, 18 Nov 2004 07:09:47 -0800

>1. IPoIB-RC and IPoIB-UC are optional.

I would prefer only one be used - either RC or UC.  I've provided some 
logic for either one as a preference but don't see a reason to have 
both.  Both just leads to options which leads to interoperability problems.

>proposal was limited to 3b.

 From an implementation point of view, all of this will be hidden within 
the driver below IP.  As such, the driver will maintain the 
associations.  Currently, each driver "instance" (may be multiple per IB 
port) will have at least 1 UD QP.  Given the existing protocol already 
defines how to share this QP with other nodes, why not just re-use it and 
avoid doing more work?  The driver can then map on a per endnode pair basis 
what *C QP go with what the UD QP and the spec remains largely silent on 
how this is accomplished.

>Do all interfaces in the IPoIB-conneced mode(CM) have the same link 

 From an implementation perspective, this is generally simplest.

>a. all are either IPoIB-RC or IPoIB-UC.

Preference is only 1 to be defined.

>         -- All unicast transmission is on the IPoIB mode i.e. RC or UC.

For a given endnode pair, the policy of which QP is used for a given 
unicast IP datagram is really a local issue.  I see some merit in the 
attempt to bifurcate this to multicast / broadcast to the UD QP and unicast 
to the *C QP.  However, if the datagram fits in the PMTU of the UD QP, then 
either could be used.  The driver would work either case.  Please keep in 
mind that multiple *C QP can be used and their usage needs to be a local 
issue and not defined within the spec.

>         symmetric. That is a good idea. Thus each peer declares the max 

Rephrase this as maximum logical MTU to avoid confusion with the IB link 
MTU.  If you start down this path, then you may need to also consider an 
exchange of what range of DiffServ code points to use as well.  Not clear 
that anyone needs to deal with any latency or bandwidth guarantees but the 
"camel's nose is starting to enter the tent" as the saying goes.

>         Local decision. Note that the peer might choose to not honour 

Agreed.

Mike

---

From: Vivek Kashyap <[email-masked]>
Date: Thu, 18 Nov 2004 11:33:40 -0800 (PST)

> both.  Both just leads to options which leads to interoperability problems.

ok.
See section 3.1 of the draft draft-kashyap-ipoib-connected-mode-02.txt. 
It states that the RC and UC are mutually exclusive flags.

> how this is accomplished.

The draft at present states that 'IPoIB-CM implementation MAY use the same UD 
QP as used by IPoIB-UD...'. See section 3.0. I believe it covers what you 
are stating.

> unicast IP datagram is really a local issue.  I see some merit in the 

Not if an implementation chooses to only receive unicast on the CM modes in
 an IPoIB-CM subnet. I think the WG must either mandate that between two
IP address all unicast communication can be over either UD or the supported CM, 
or state that all unicast communication must be over IPoIB-CM. Hence my
attempt at a detailed discussion on these issues.

Issues such as in order delivery need to be considered: e.g. if RC and UD are 
used to mix up the traffic, say of TCP segments of the same connection, they 
may no longer be received in order.

> Rephrase this as maximum logical MTU to avoid confusion with the IB link 

It is covered in section 5.1 of the draft. 

> "camel's nose is starting to enter the tent" as the saying goes.

The camel comes along if  Diffserv  etc. as listed above are 
included. Hence they are not in the draft.

---

From: Michael Krause <[email-masked]>
Date: Thu, 18 Nov 2004 13:26:43 -0800

>It states that the RC and UC are mutually exclusive flags.

My preference is to only support one of the two in a spec not to have flags 
to indicate what is implemented.  The benefits of connected mode operation 
should be done with only one form of communication not two.

> > >As far as I know, there has been an agreement since the earliest 
> connected
> > >mode
> > >draft I posted.
> > >
> > >

If a designer is stupid, they may do this. However, one would expect some 
intelligence here and one may prefer to have specific data flows or 
DiffServ code points or whatever used to determine which connection or 
which UD QP and that one would again apply an intelligent and predictable 
algorithm such that mix-n-match for a given TCP connection does not 
occur.  Given multiple *C QP can be supported, it is not tenable to state 
that all unicast must go over a given QP or that no unicast can occur on a 
UD QP.

> joined
> (i.e. at

---

From: Vivek Kashyap <[email-masked]>
Date: Thu, 18 Nov 2004 16:12:23 -0800 (PST)

> should be done with only one form of communication not two.

A given subnet will support only one of the two. Not both simultaneously. The 
flag only indicates which type it is. RC and UC are both useful to different 
people and implementations so both are allowed. I suggest that both not be 
allowed in the same IPoIB subnet though.

> that all unicast must go over a given QP or that no unicast can occur on a 

You mised my point which was that the specification cannot be silent on this 
and say it is a local issue. That can lead to interoperability failure. The 
specification must support or disallow  unicast communication over UD QP in an 
IPoIB-CM.  

You prefer that such communication be supported. That works. Any other thoughts?

---

From: Michael Krause <[email-masked]>
Date: Thu, 18 Nov 2004 16:22:35 -0800

>allowed in the same IPoIB subnet though.

To be explicit, I think there is benefit in implementing one and only one 
of the two.  Having two options serves no purpose and adds unnecessary 
complexity.  Interoperability will end up requiring both to be done if 
customers are to not get upset.  Let's just pick one of the two and apply 
KISS.  To get this started, I'll propose RC as that is a bit nicer to the 
fabric than UC and is already implemented in most OS and CA drivers today 
so it makes it faster to adopt with minimal driver software update.

<snip>

>You prefer that such communication be supported. That works. Any other 

I prefer that guidance be provided and that it remain a local 
implementation issue as to what QP is used for a given flow.  I do not see 
interoperability issues only potential performance if people are 
stupid.  The industry has a way to deal with stupidity and too much time is 
spent on preventing people from being stupid.  Even a so-so intelligent 
implementation could have a simple flag for a given target IP address that 
states which QP to target for all or a subset of the flows with minimal 
cost to implement and troubleshoot / validate.

Mike

---

From: Vivek Kashyap <[email-masked]>
Date: Thu, 18 Nov 2004 17:14:22 -0800 (PST)

> >allowed in the same IPoIB subnet though.

RC and UC both have benefits. There is almost no difference other than
the connection flag between the two. 

> cost to implement and troubleshoot / validate.

If something is left unspecified there is every chance that incompatible
implementations result - this has nothing to do with the mental faculties
of the implementors. Therefore I'll add relevant text.

---

From: Michael Krause <[email-masked]>
Date: Thu, 18 Nov 2004 17:45:09 -0800

>the connection flag between the two.

Many host OS implementations do not support UC as RC and UD are all that is 
really required within the industry.  The ACK overhead associated with RC 
is truly noise and the end-to-end credits are very nice as IB now supports 
three signaling rates combined with 4 link widths (though only three are 
really being implemented).  Such a permutation in bandwidth capability 
makes RC a more tenable / good citizen as we designed it to be so I'd 
prefer RC.

>of the implementors. Therefore I'll add relevant text.

One can just provide an implementation note to avoid any mental short 
comings and still avoid specifying this.  That should be sufficient while 
maintaining KISS.

Mike