Subject: RE: [Ipoverib] draft-ietf-ipoib-ip-over-infiniband-00.txt

From: "Larsen, Roy K" <[email-masked]>
Date: Fri, 22 Feb 2002 17:46:35 -0800

> -----Original Message-----
> From: Vivek Kashyap [mailto:[email-masked]]
> Sent: Friday, February 22, 2002 3:38 PM
> To: [email-masked]
> Cc: [email-masked]; [email-masked]
> Subject: Re: [Ipoverib] draft-ietf-ipoib-ip-over-infiniband-00.txt
> 
> 
> > 
> > Vivek,
> > 
> > I think you've done a good job outlining the possible 
> approaches and their
> > ramifications.  I have only two minor nits.  It wasn't 
> clear to me if you
> > were stating that methods (ii) and (iii) would require 
> using separate QPs
> > for IPv4 and IPv6.  If so, I believe an implementation 
> could choose to use a
> > single QP for all IPoIB protocols and remain completely 
> interoperable with
> > all other implementations.  In which, case Jerry's 
> observation is still
> > valid.
> 
> [Which observation of Jerry ? Or, do you mean the comments in the mail
> I appended (was from Akihiko).]

Sorry, I meant to refer to Akihiko

> 
> If we use (ii) or (iii) we can demultiplex the protocols on the same
> QP. It is therefore completely upto the implementation to share or
> not-share the QPs.
> 
> So, in either case we drop the recommendation in the draft.
> 

Thanks for clearing that up for me.

> > 
> > Second, I don't think there needs to be a concern about 
> using a type field
> > in the face of arbitrary protocols.  Protocols can only 
> send to a QP that
> > they have obtained the QPN and Q_Key for.  This would 
> presume that one would
> > not advertise the QP/Q_KEY to protocols that the 
> implementation is not
> > prepared to handle.  That said, I would agree that if the 
> consensus is to
> > encapsulate type, using the existing Ethernet types makes sense.
> 
> Not really. If the desire to use 'other' extends to arbitrary
> protocols, then those protocols need to agree on the format i.e. agree
> to use the type. With limiting the 'other' protocols to ethertype we
> are talking about protocols that agree to use the type field and also
> agree on which type value belongs to whom.
> 
> May be I am confused but the following bothers me - it is an
> interoperability issue:
> 

First, let me make sure everyone knows where I stand.  My support for
encoding an LSAP has nothing to do with enabling non-IPoIB protocols.

> On the whole, the 'other' protocols over IB is a weak argument since
> we are then opening the IP QPN (received by IP address resolution) to
> exchange non-IP protocols.

My point is that 'we' (the IPoIB WG) are not opening up the IPoIB QP. The
best the IPoIB RFC can do is state how a compliant implementation discovers
and advertises the QP/Q_Key it wants to receive IPoIB frames over and the
format and semantics of those frames.  It can not restrict an implementation
from allowing other forms of data from using that same QP.  

> 
> Non-IP protocols sent to IP address ? or, proprietary headers for some
> load-balancing/other solution sent to an IP address (and QPN)? How
> would implementation X talk to Y which ends up encapsulating
> differently or how do they decide to agree on the encapsulation? Do we
> end up in the situation of specifying a SNAP/NOSNAP equivalent on
> IPoIB interfaces? These other types should get a QPN of their own -
> they are not connected to IP at all.

The target system certainly would need to assign a QP and if it felt it
could handle it, there is nothing the IPoIB RFC can do to prevent it from
being the same QP that the implementation used for IPoIB.

> 
> An IP QPN is not a link like other media - it is purely an IP only
> link or at a maximum an IP/ARP link. So, such non-IP types really
> don't help. This seems to suggest that we need to state that the types
> are :IPv4,IPv6 and ARP/RARP.
> 
> But if we are limited to just these then why not just use the first
> few distinguishing bits (which serve the same purpose) and avoid
> handling a pseudo-header?

I think that is exactly what Akihiko wants to do; use a QP as a link/media
interface.  But to be honest, that is really how we are using it as
demonstrated by our ARP mechanism.  I would venture to say that many
implementations will use a single QP for all IPoIB traffic which is as close
to using the QP as a virtual link as you can get (within the scope of
IPoIB).  I also think its a pretty safe bet that most implementations will
be integrated into the OS as a link driver.  We can embrace these realities
and enable traditional layering by encapsulating the LSAP provided by the
link driver user, or we can look into the payload and save a few bytes.  

Clean layering has always been my concern and it should be clear to this
list that it is purely philosophical.  The fact that encasulating the LSAP
will also enable Akihiko just validates the virtues of a pure network
architecture.

Roy

> 
> 
> Vivek
> 
> > 
> > Roy
> > 
> > -----Original Message-----
> > From: Vivek Kashyap [mailto:[email-masked]]
> > Sent: Friday, February 22, 2002 12:25 PM
> > To: [email-masked]
> > Subject: Re: [Ipoverib] draft-ietf-ipoib-ip-over-infiniband-00.txt
> > 
> > 
> > > 
> > > One Comment about Vivek/Jerry's Feb.6 draft.
> > > 
> > > >> >The QPs advertised for IP
> > > >> >         communication MUST NOT be used for other protocols.
> > > >> 
> > > >> It is unclear if this can be enforced or worthwhile to 
> restrict so why 
> > > >> state it.  If used for anything else, this is a local 
> implementation 
> > > >> problem to resolve.
> > > >
> > > >        Yes, it was discussed and agreed (I think) that 
> we will use
> > > >        SHOULD. 
> > > 
> > > I think this sentence should be deleted.
> > > Because I think it is not recommendation but 
> implementation-dependent that
> > >  if QP(s) for IP is shared with others or not.
> > > 
> > 
> > This sentence is related to the comment in the draft about resolving
> > the encapsulation format. We should try and resolve it in the coming
> > 10 days or so (March, 2nd is the cut-off for drafts).
> > 
> > I present the issues below and hopefully enough people will 
> join in to
> > allow the chair to call a consensus.
> > 
> > MUST is equivalent to the IBA model: --- (i)
> > 
> >         The IB architecture identifies data streams based on the QP
> >         that is associated with them. It does not include a type
> >         field.  
> > 
> >         This reasoning would suggest (say) we use a unique QP for
> >         IPv6. This QP (IPv6_QP) would be the one resolved during ND
> >         and is therefore expected to send and receive only IPv6
> >         packets.
> > 
> >         Thus the protocols can be directly encapsulated in the IB
> >         payload. 
> > 
> >         This model therefore makes it a MUST to separate the data
> >         streams using the QPs.
> > 
> > 
> > The above model can be applied to all possible protocols on IB. Thus
> > we need to have a separate QP for IPv6 and a separate one for IPv4.
> > This is what was suggested in the draft with the caveat that the
> > section will be effected based on the resolution of the
> > 'type'/encapsulation issue.
> > 
> > We get a separate QPN using ND for IPv6. Get one for IPv4 using ARP.
> > Now, we could choose to return two (one for IPv4 and one for ARP).
> > 
> > As far as I can tell the choice is to exchange only one QPN for IPv4
> > and related protocols (ARP and its op-codes).
> > 
> > Thus we are now called upon to distinguish between these 
> two types of
> > packets. There are two ways to do this:
> > 
> > a) Protocol signature bits  --- (ii)
> > 
> >    The QP is filtering the packets to just the two 
> protocols. The two
> >    protocol headers are sufficiently different to separate the
> >    protocols by considering the first bits in the protocols as
> >    defining a 'type' field.
> > 
> >    The driver thus checks the type to determine the upper layer but
> >    does not 'step over' the field when it passes the packet to the
> >    upper layer.
> > 
> > b) A 'type' in the payload --- (iii)
> > 
> >    One can add a type field in the payload that in turn 
> indicates the
> >    encapsulated protocol. 
> > 
> > So, we can do one of (i), (ii) or (iii). All three will work. Of the
> > three, (ii) and (iii) are the better options if we want to keep the
> > QPN exchange (in the case of IPv4, IPv6 is unaffected) 
> limited to only
> > one. One could define a separate encapsulation for IPv4 and v6 but I
> > doubt if many are keen on that.
> > 
> > That leaves the solution to (ii) or (iii). 
> > 
> > With (ii) we keep ourselves limited to the IP specific protocols -
> > such as IPv6/IPv4/ARP. This is really the scope of IPoIB and so is
> > fine. 
> > 
> > However, some people have voiced the opinion that they would really
> > like to use 'other' protocols over the same QP. This can be
> > accomplished by adding a 'type' in the payload.
> > 
> > This solution of course depends on the meaning of 'other'. If it is
> > arbitrary protocols then there is no good solution since 
> the arbitrary
> > protocols may follow their own rules and hence clash with any
> > encapsulation scheme we choose.
> > 
> > We could alternatively say that the ethernet numbers for
> > IPv6/IPv4/ARP/RARP will be used for the respective encapsulation. We
> > don't restrict the use of other values - but the 'others' are 
> > restricted to ethernet numbers. An IPoIB interface driver, 
> of course,
> > is free to reject anything other than IPv6/v4/ARP/RARP.
> > 
> > I am ok with either of (ii) or (iii). Depending on the consensus
> > (hopefully called in the next few days) the draft can be updated and
> > submitted by March 2nd.
> > 
> > 
> > Vivek
> > -- 
> > Vivek Kashyap
> > Linux Technology Center, IBM
> > [email-masked] 
> > [email-masked] 
> > 503 578 3422 (o)
> > 
> > _______________________________________________
> > IPoverIB mailing list
> > [email-masked]
> > https://www1.ietf.org/mailman/listinfo/ipoverib
> > 
> > _______________________________________________
> > IPoverIB mailing list
> > [email-masked]
> > https://www1.ietf.org/mailman/listinfo/ipoverib
> > 
> > 
> 
> 
> -- 
> Vivek Kashyap
> Linux Technology Center, IBM
> [email-masked] 
> [email-masked] 
> 503 578 3422 (o)
> 

_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib

---

From: [email-masked]
Date: Mon, 25 Feb 2002 23:23:57 +0900

>From: "Larsen, Roy K" <[email-masked]>
>Subject: RE: [Ipoverib] draft-ietf-ipoib-ip-over-infiniband-00.txt
>Date: Fri, 22 Feb 2002 17:46:35 -0800
>

# Oh, does it ? I am a litle surprised, Roy, But, I give some comments here.

>from allowing other forms of data from using that same QP.  

Agree.

>being the same QP that the implementation used for IPoIB.

That's right. It is what I'd like to say.

>interface. 

Exactly!
In network stacks-layered architecture in general OSes
,IMO, a QP is a single interface (i.e. a link)
and using multiple QPs is the best suitable for load-balancing
and parallel processing, especially if different VLs and CPUs can be used.
In our case, defining multiple IPoIB interfaces (: one QP per IPoIB interface)
 is the case.

> But to be honest, that is really how we are using it as
>IPoIB).

> I also think its a pretty safe bet that most implementations will
>be integrated into the OS as a link driver.

Yes. I think so.

>  We can embrace these realities
>list that it is purely philosophical.

I believe simplicity is the most important and it gives us clarity,
 consistency, and generality.
In this argument, my choice is not "a QP per protocol" (multiple QPs per
 interface) but "a QP per interface".
i.e. a QP is an interface, a IPoIB datalink driver.
It fits layered network stacks architecture and IPoIB driver as a "datalink-
layer driver" very well, I also think, as Roy says above.

# I think using multiple QPs (by using multiple interfaces) are
# the best suitable for load-balancing and parallel processing.

>  The fact that encasulating the LSAP

It may be right, too... I understand what "philosophical issue" means
 a little..

Thanks,
Akihiko