Subject: RE: [ipoverib] reasons for not using LID ?

From: "Arvind Murching" <[email-masked]>
Date: Thu, 17 Jan 2002 17:33:46 -0800

Below...

-----Original Message-----
From: Yaron Haviv [mailto:[email-masked]] 
Sent: Thursday, January 17, 2002 5:31 PM
To: Arvind Murching; [email-masked]; [email-masked]
Subject: RE: [ipoverib] reasons for not using LID ?

My remarks attached

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]On Behalf
Of
Arvind Murching
Sent: Friday, January 18, 2002 2:57 AM
To: [email-masked]; [email-masked]
Subject: RE: [ipoverib] reasons for not using LID ?

The argument should be the other way around, i.e. fields need to justify
their necessity to be part of a protocol.
[YH] I think many arguments where already raised, now it's the
opposition
turn to put their arguments
[Arvind] The need for a redundant field is not obvious.

Some reasons why not:

- LID values cannot be used across IB subnet boundaries
[YH] you would any way need to store both LID (Router) and GID (node) in
this case somewhere, you wouldn't go to the SM for any new IP packet
wont
you
the routers in this case will any way need to change the LID in the ARP
packets (Proxy ARP and all that)
I believe this case is the one that benefits most from the LID, avoiding
the
long and complex multi-subnet resolution process
[Arvind] Consider the most general case of an IB router routing between
IB subnets. This router will not modify an ARP reply packet to insert
its LID into the payload. However, it will insert its SLID into the IB
header of packets that it forwards. Proxy ARP boxes can do the same,
too.

- A LID cannot be used as part of a static ARP entry
[YH] We didn't say not to have GID, you could still put GID as a static
entry

- The SLID is already present in case someone needs it
[YH] also regular ARP has it, so why not follow that path
[Arvind] This is not an argument for adding redundant fields.

My 2c
- Arvind

-----Original Message-----
From: Yaron Haviv [mailto:[email-masked]]
Sent: Thursday, January 17, 2002 4:47 PM
To: [email-masked]
Subject: [ipoverib] reasons for not using LID ?

By now it is obvious that there are the ones that wants LID's in the ARP
packets

Could the ones that don't give their reasons for why not to
Not just things like "you don't need it", "we want to save 16-bit", etc'
But, reasons for what problems could arouse from having it in the packet
I haven't heard any real argument that would make me consider removing
it
until now
we then will review them and see if it is possible to deal with those
cases,
if not I personally would drop my support in LID's

If people want it for their reasons, the ones that don't also need to do
some explaining

Yaron

_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib

---

From: "Bill Strahm" <[email-masked]>
Date: Thu, 17 Jan 2002 18:09:57 -0800

I am very confused, why will an IB router have to modify an IP ARP packet at
all.  If there is something in the spec that REQUIRES that I think I would
be in favor of ripping it out and fixing this...

Consider IB routers as looking like Ethernet switches, it is just a layer 2
device that sends packets to the correct collision domain.  I see that as
what an IB router is for the IP layer on top of it.  You would have a hard
time convincing me that you want the router sniffing and modifing ARP
packets.  This would be a huge HUGE layer violation.

Now here is your arguement for not including LIDs in the ARP response, I
want a single ARP response for everyone that sends me an ARP request in my
IP subnet.  If this IP subnet crosses an IB subnet, this LID will not be
valid (or the router has to munge it for me)  This seems a lot of work to
get past the case of a simple lookup to the SM that the IBTA Architecture
would have you do when you are communicating with a new endpoint.

Bill

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]On Behalf
Of Yaron Haviv
Sent: Thursday, January 17, 2002 5:46 PM
To: 'Arvind Murching'; [email-masked]
Subject: RE: [ipoverib] reasons for not using LID ?

Arvind,

The only real argument you make is implementation in Routers
Since we build such boxes, I could tell you dealing with an ARP in the
router in case of IP span IB is complex anyway
Believe me, the simplification you are offering me creates more work than
help inside the router
As you also stated in the passed there are also very little amount of ARP
packets, so replacing it in one or two paces doesn't change anything

Yaron

-----Original Message-----
From: Arvind Murching [mailto:[email-masked]]
Sent: Friday, January 18, 2002 3:34 AM
To: [email-masked]; [email-masked]
Subject: RE: [ipoverib] reasons for not using LID ?

Below...

-----Original Message-----
From: Yaron Haviv [mailto:[email-masked]]
Sent: Thursday, January 17, 2002 5:31 PM
To: Arvind Murching; [email-masked]; [email-masked]
Subject: RE: [ipoverib] reasons for not using LID ?

My remarks attached

-----Original Message-----

Some reasons why not:

My 2c
- Arvind

-----Original Message-----
From: Yaron Haviv [mailto:[email-masked]]
Sent: Thursday, January 17, 2002 4:47 PM
To: [email-masked]
Subject: [ipoverib] reasons for not using LID ?

Yaron

---

From: Yaron Haviv <[email-masked]>
Date: Fri, 18 Jan 2002 04:37:51 +0200

Bill, the answer to your first question relays in your second one

When IP spans IB, the router receive a packet with the local LID of the
first subnet (even when it is just in the S/DLID) and needs to modify it to
a packet with a local LID of the target subnet
Otherwise it won't get to the target, or returned to the sender
I would be glad to know what is your trick for avoid that
To avoid loops the router will need to implement a proxy ARP, which
maintains the virtualized routing tree
Looking at it like a layer 2 switch is to much of a theoretical abstraction

Answering the 2nd Q, by now you see that the ARP been sent to the target has
the router LID, and when you will reply it will get just to the right place
The end node operation doesn't change in any case

Yaron

-----Original Message-----
From: Bill Strahm [mailto:[email-masked]]
Sent: Friday, January 18, 2002 4:10 AM
To: [email-masked]; 'Arvind Murching'; [email-masked]
Subject: RE: [ipoverib] reasons for not using LID ?

Bill

-----Original Message-----

Arvind,

Yaron

-----Original Message-----
From: Arvind Murching [mailto:[email-masked]]
Sent: Friday, January 18, 2002 3:34 AM
To: [email-masked]; [email-masked]
Subject: RE: [ipoverib] reasons for not using LID ?

Below...

-----Original Message-----
From: Yaron Haviv [mailto:[email-masked]]
Sent: Thursday, January 17, 2002 5:31 PM
To: Arvind Murching; [email-masked]; [email-masked]
Subject: RE: [ipoverib] reasons for not using LID ?

My remarks attached

-----Original Message-----

Some reasons why not:

My 2c
- Arvind

-----Original Message-----
From: Yaron Haviv [mailto:[email-masked]]
Sent: Thursday, January 17, 2002 4:47 PM
To: [email-masked]
Subject: [ipoverib] reasons for not using LID ?

Yaron

---

From: Yaron Haviv <[email-masked]>
Date: Fri, 18 Jan 2002 03:46:21 +0200

Arvind,

Yaron

-----Original Message-----
From: Arvind Murching [mailto:[email-masked]]
Sent: Friday, January 18, 2002 3:34 AM
To: [email-masked]; [email-masked]
Subject: RE: [ipoverib] reasons for not using LID ?

Below...

-----Original Message-----
From: Yaron Haviv [mailto:[email-masked]]
Sent: Thursday, January 17, 2002 5:31 PM
To: Arvind Murching; [email-masked]; [email-masked]
Subject: RE: [ipoverib] reasons for not using LID ?

My remarks attached

-----Original Message-----

Some reasons why not:

My 2c
- Arvind

-----Original Message-----
From: Yaron Haviv [mailto:[email-masked]]
Sent: Thursday, January 17, 2002 4:47 PM
To: [email-masked]
Subject: [ipoverib] reasons for not using LID ?

Yaron

---

From: "Bill Strahm" <[email-masked]>
Date: Thu, 17 Jan 2002 20:28:54 -0800

I do not agree with your analysis in your first answer.  Yes the IB router
would receive the packet on its first interface and forward it on its second
interface replacing the DLID with its own DLID.  This is what a layer 3
router does.

Now look at the Infiniband fabric as a layer 2 for the IP layer 3.  At this
point you want layer 2 semantics.  At this point IB routers look like layer
2 switches and MUST NOT interfere with the layer 3 packets on top (otherwise
it is a layer 3 router, and it might as well do IP routing and we don't have
the subnet spanning problem at all)

The job of an IB router in an infiniband fabric is much simpler if we do not
have LID in the ARP packet at all, and just have the destination node poll
the SM for the LID based on the GID.

Bill

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]On Behalf
Of Yaron Haviv
Sent: Thursday, January 17, 2002 6:38 PM
To: 'Bill Strahm'; [email-masked]
Subject: RE: [ipoverib] reasons for not using LID ?

Yaron

-----Original Message-----
From: Bill Strahm [mailto:[email-masked]]
Sent: Friday, January 18, 2002 4:10 AM
To: [email-masked]; 'Arvind Murching'; [email-masked]
Subject: RE: [ipoverib] reasons for not using LID ?

Bill

-----Original Message-----

Arvind,

Yaron

-----Original Message-----
From: Arvind Murching [mailto:[email-masked]]
Sent: Friday, January 18, 2002 3:34 AM
To: [email-masked]; [email-masked]
Subject: RE: [ipoverib] reasons for not using LID ?

Below...

-----Original Message-----
From: Yaron Haviv [mailto:[email-masked]]
Sent: Thursday, January 17, 2002 5:31 PM
To: Arvind Murching; [email-masked]; [email-masked]
Subject: RE: [ipoverib] reasons for not using LID ?

My remarks attached

-----Original Message-----

Some reasons why not:

My 2c
- Arvind

-----Original Message-----
From: Yaron Haviv [mailto:[email-masked]]
Sent: Thursday, January 17, 2002 4:47 PM
To: [email-masked]
Subject: [ipoverib] reasons for not using LID ?

Yaron

---

From: Yaron Haviv <[email-masked]>
Date: Fri, 18 Jan 2002 14:53:08 +0200

Bill,

First when forwarding packets from one subnet to another, the LRH header
needs to be reconstructed
This involved replacing both the DLID & SLID in the new header, so we do
deal with the packet
And we do have the LID'd in the packet LRH header anyway
And we don't keep the same LID's in a way that the destination will get a
LID which belongs to a different subnet

You assume that the IB routers works like a layer 2 Ethernet switch
Unfortunately this is not the case, since as mike noted I couldn't reveal
the content of the LWG discussions I cannot go into it
Lets just say as one of the people active in the router sub-group that it is
totally different and a lot of things still needs to be determined
I would choose at least initially to implement something like Proxy ARP to
deal with the Issues
One difference I could point out is that the Ethernet spanning tree is a
self-learning mechanism where as IB routing is not
The Proxy ARP mechanism compensates for that fact

I'm trying to stay away from the theory and see how today I would implement
those things the most efficient way
When digging into it and going over all the potential problems, I would
personally like to avoid the SM and especially when going to a different
subnet
And not just because of the few extra milliseconds delay, and SM overhead,
but because I know it could work in all the scenarios

One thing worth digging into is how to enable avoiding the SM using the SLID
and other extra information that will enable detecting when that LID is
invalid, plus maybe the Valid LID, this way if/when IB routing will be
different it might answer your desires too

Yaron

 From: [email-masked] [mailto:[email-masked]]On Behalf Of
Bill Strahm
Sent: Friday, January 18, 2002 6:29 AM
To: [email-masked]; [email-masked]
Subject: RE: [ipoverib] reasons for not using LID ?

Bill

-----Original Message-----

Yaron

-----Original Message-----
From: Bill Strahm [mailto:[email-masked]]
Sent: Friday, January 18, 2002 4:10 AM
To: [email-masked]; 'Arvind Murching'; [email-masked]
Subject: RE: [ipoverib] reasons for not using LID ?

Bill

-----Original Message-----

Arvind,

Yaron

-----Original Message-----
From: Arvind Murching [mailto:[email-masked]]
Sent: Friday, January 18, 2002 3:34 AM
To: [email-masked]; [email-masked]
Subject: RE: [ipoverib] reasons for not using LID ?

Below...

-----Original Message-----
From: Yaron Haviv [mailto:[email-masked]]
Sent: Thursday, January 17, 2002 5:31 PM
To: Arvind Murching; [email-masked]; [email-masked]
Subject: RE: [ipoverib] reasons for not using LID ?

My remarks attached

-----Original Message-----

Some reasons why not:

My 2c
- Arvind

-----Original Message-----
From: Yaron Haviv [mailto:[email-masked]]
Sent: Thursday, January 17, 2002 4:47 PM
To: [email-masked]
Subject: [ipoverib] reasons for not using LID ?

Yaron

---

From: "Bill Strahm" <[email-masked]>
Date: Fri, 18 Jan 2002 08:42:10 -0800

Question for you Yaron,

For you ARP Proxy (on the IB Router)...

How will you detect ARP packets, given that the ARP response is Unicast back
to the sender.  The response can be on any QPN going to any destination.  I
would be scared of using pattern matching because then it starts to
potentially interfere with other protocols that are sending random binary
data, and also that would require a filter on every packet coming across the
link (at 12x rates even)

This is a hard problem that can easily be solved by a simple GID->LID lookup
at the SM.  Either that or limit IP subnets to a single IB subnet and do IP
routing to cross IB subnets, then the IB router just has to handle IP
routing which is a simple problem especially in a static environment.

Bill

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]On Behalf
Of Yaron Haviv
Sent: Friday, January 18, 2002 4:53 AM
To: 'Bill Strahm'; [email-masked]; 'Libor Michalek'
Subject: RE: [ipoverib] reasons for not using LID ?

Bill,

Yaron

Bill

-----Original Message-----

Yaron

-----Original Message-----
From: Bill Strahm [mailto:[email-masked]]
Sent: Friday, January 18, 2002 4:10 AM
To: [email-masked]; 'Arvind Murching'; [email-masked]
Subject: RE: [ipoverib] reasons for not using LID ?

Bill

-----Original Message-----

Arvind,

Yaron

-----Original Message-----
From: Arvind Murching [mailto:[email-masked]]
Sent: Friday, January 18, 2002 3:34 AM
To: [email-masked]; [email-masked]
Subject: RE: [ipoverib] reasons for not using LID ?

Below...

-----Original Message-----
From: Yaron Haviv [mailto:[email-masked]]
Sent: Thursday, January 17, 2002 5:31 PM
To: Arvind Murching; [email-masked]; [email-masked]
Subject: RE: [ipoverib] reasons for not using LID ?

My remarks attached

-----Original Message-----

Some reasons why not:

My 2c
- Arvind

-----Original Message-----
From: Yaron Haviv [mailto:[email-masked]]
Sent: Thursday, January 17, 2002 4:47 PM
To: [email-masked]
Subject: [ipoverib] reasons for not using LID ?

Yaron

---

From: Vivek Kashyap <[email-masked]>
Date: Fri, 18 Jan 2002 10:22:17 -0800 (PST)

Bill, Yaron,

	There are two situations 
		- proxy-ARP for a node on another subnet
		- span IP subnets across IB subnets

1. 	proxy-ARP for a node on another subnet works very easily. 
        You return the LID of the local IB router and the GID of the
        remote IB node. Note this is possible with the current
        specification. If only the GID is included excluding the LID
        this is not possible since you can't lookup the remote node's
        GID with the SA. You need to include both. [ SA to SA
        coordination within an IB subnet isn't specified what of
        across the subnet. ]

        Note that the ARP packet is fielded by the IPoIB device
        logically/physically associated with the IB router. The ARP
        packet itself does not travel beyond the IB subnet.

2. 	For IP subnets to span IB subnets there is too much 
        speculation and no justification. First of all we don't know
        why we need it. We don't know when it will be possible. We
        don't know if it will be possible. It is a complicated issue -
        too many things have to be updated in IB architecutre.

        Even for argument's sake lets take it that it was possible for
        IP subnets to span IB subnets. The solution is very simple
        when LID is included. And it is compatible with the solution
        in 1. above.

        The receiving node's ARP module looks at the GID-prefix. If
        the prefix is of another subnet then the routing
        infrastructure is looked up and the router's LID is recorded
        otherwise the LID in the packet is recorded.

I find it odd that we are discussing 'IP subnets spanning IB subnets'
but not proxy-ARP that can work, happen and maybe is all that will
ever be needed or possible(if IBTA doesn't solve routing/cross subnet
issues). 

Vivek

-- 
Vivek Kashyap
Linux Technology Center, IBM
[email-masked] 
[email-masked] 
503 578 3422 (o)

---

From: Michael Krause <[email-masked]>
Date: Mon, 21 Jan 2002 12:40:34 -0800

>         packet itself does not travel beyond the IB subnet.

Within the requester's IB subnet, there will exist at least one endnode 
that will respond on behalf of any request that targets an endnode that 
exists on another IB or IP subnet.   The responder will work with the SA to 
determine the IB router; an IP router would respond on behalf of its 
responsibility "cloud" and it too would work with the SA to determine the 
IB LRH values to use to establish appropriate communication between the 
requester and the IP router.  In both cases, the responder is responsible 
for determining the appropriate LRH / GRH (where applicable) header 
parameters to establish the communication channel.

>         too many things have to be updated in IB architecutre.

The justification is that vendors and customers believe it is the right 
thing to do and there is both technical and economic feasibility to achieve 
this objective.  Unless there is strong consensus to not support this type 
of functionality, the workgroup must take steps to enable it or at least 
not preclude this from being implemented at some future date once the IBTA 
has provided the appropriate specification updates.

>ever be needed or possible(if IBTA doesn't solve routing/cross subnet

I believe Paul provided a good starting position on a compromise a few days 
back.  Why don't people debate that position instead of repeating the same 
arguments on either side.

Mike

---

From: Michael Krause <[email-masked]>
Date: Mon, 21 Jan 2002 11:46:01 -0800

>needs to be reconstructed

 From a logical perspective, the LRH is stripped off by the ingress IB 
router port and added by the egress router port.  LRH construction and GRH 
updates are described in 19.2.4 volume 1, version 1.0.a.  The primary 
fields impacted by crossing IB subnet boundaries are: DLID, SLID, SL, VL, 
and Hop Count.  There is no statement concerning the GRH flow label but 
this is also subject to modification.

>the content of the LWG discussions I cannot go into it

An IB router from an IP perspective layer 2 logical perspective is no 
different than an Ethernet switch. This was noted at the first IETF BoF 
last Spring when this issue was raised and one person stated that 
individual IB subnets can be treated as point-to-point links from an IP 
perspective for much of the issues being raised.  There is nothing within 
the current specification or from an IB architecture perspective that would 
preclude someone from treating them as such for IP purposes.  IB routers do 
not know anything about what they carry within their payload and thus are 
not aware whether the traffic is IP, ARP, Application, etc.  They only know 
parse the GRH to determine the longest prefix match and combine this with 
other fields within the GRH to determine the egress port, update the GRH, 
and build a new LRH.

>The Proxy ARP mechanism compensates for that fact

An IB router is not an IP router (though an implementation may support 
both) and as such the proxy ARP mechanism is only applicable to an IP 
router - it has no bearing on the IB router protocol or the IB router 
responsibilities.

>but because I know it could work in all the scenarios

The IB router must interact with each subnet SM to understand its LID, LMC, 
VL, link width, etc.  The IB router must interact with each subnet SA to 
understand all of the destination endnodes that it is allowed to reach.  It 
must also interact with the SA to understand if there are any other IB 
routers attached to this subnet so that the router protocol can establish 
the appropriate router tables, etc. between these two ports.  All of this 
has been described in the various presentations made by the IBTA as well as 
the current specification.  I do not see how the router can avoid 
communication with the SM for fabric-specific values and with the SA for 
path record determination.  I expect the router to learn most of this 
information during its initialization steps and then update as required to 
deal with fabric event changes.  I do not see the need for the router to 
impose any serious performance requirements on the SM or the SA.

>different it might answer your desires too

Again, I don't see how one avoids the SM or the SA if each local subnet's 
switching / QoS policies are to be complied with to ensure the fabric 
remains operational.

Mike

---

From: Vivek Kashyap <[email-masked]>
Date: Tue, 22 Jan 2002 15:15:24 -0800 (PST)

>parameters to establish the communication channel.

Is the solution then that the remote subnet's GIDs not only have to be
known to the IPoIB device but also recorded in the SA/SM or at least
the remote GID-prefix is recorded with the SA/SM ?

My note above describes a simpler solution that merely takes the LID and
the GID from the ARP reply and the SA/SM need not be involved. If the
implementation prefers to get the SL/path rate from the SA (for the IB
router) it gets them based on the LID received (which is that of the
IB router). The solution is much simpler. Any IPoIB system on the
subnet can answer for the remote node and the packet will reach
through the router with no setup required.

The suggestion that the LID from the LRH be used works efficiently but
only if the responder is the IB router itself. If another device
(IPoIB device) answers for it then it will, as has been suggested,
have to indicate that the LID in the LRH is not the right one and the
solution will default to the more complex 'record non-local GIDs in
the SA'.

Is there a query/set of queries that returns the routers in the subnet and
that tells which routers to choose ? Also, if a non-local GID prefix
is used can one determine the router to use ?

If so then the solution 'use LID in LRH but only if ARP/ND didn't mark
it invalid' can work. If such a 'non-local GID' solution will not work
now because the routing infrastructure/SA records that are needed
don't exist and one has to await IB routing then the invalid flag is
not needed either. Or, if in general such an across the subnet
proxy-ARP solution can be 'ignored' then too we don't need the invalid
flag. Using the LID from LRH is sufficient.

As far as I know 'non-local GID/associated router(s)' records don't
exist and have to be invented. I would rather that we have a solution
that works now and so support LID+GID. However, if we 'ignore' this
one case of proxy-ARP across IB-subnet then LID from LRH will do.

Question 1: Can the kind of proxy-ARP described here be implemented if
the LID is not included in the ARP/ND ? i.e. do the above queries and
records exist in the SA database? If not, we either need to say that
such a solution is not possible or define a solution.

Question 2: Is the LRH:LID always available for all UD communication ? Is
it equally applicable to receiving UD packets on xCAs, IB routers and IB
switches (port 0 probably) ? If not, then when is it not applicable. (I do
expect it to be available in all cases but want to be sure).

>has provided the appropriate specification updates.

The reason I am asking for such justificaiton because it is being used
to argue for various solutions. It is better to know what is the
reason for wanting it and then we can decide if we still want to go
ahead or not. Till now no real reason has been specified. We create IB
subnets for isolation, management etc. then why join them back again
in IPoIB subnets. Wouldn't IP subnetting serve the purpose anyway ?

Vivek

---

From: Yaron Haviv <[email-masked]>
Date: Wed, 23 Jan 2002 02:02:18 +0200

Vivek,

As one of the guys suggesting the Proxy ARP approach, I could answer your
first question
Yes, the LID in the LRH will do, no need for LID in the ARP
I just need the Valid bit mainly for efficiency in multi subnet
configurations

I also assume regarding your second question, LID is always there in UD

I believe that we could remove the LID from the ARP, and not harm any
possible configuration
And as one of the supporters for that approach in the pass, I double checked
it before making the decision

So, lets agree on this subject and move forward with the next topic

Yaron

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]On Behalf Of
Vivek Kashyap
Sent: Wednesday, January 23, 2002 1:15 AM
To: Michael Krause
Cc: [email-masked]
Subject: Re: [ipoverib] reasons for not using LID ?

Vivek

---

From: Vivek Kashyap <[email-masked]>
Date: Tue, 22 Jan 2002 18:31:29 -0800 (PST)

> Yes, the LID in the LRH will do, no need for LID in the ARP

Yes, it will work. No issues with it generally.

-----------

But proxy-ARP across the IB subnet has been on the desired list of
some people - you have mentioned it too. What I am pointing out is
that such proxy-ARP won't work. This is because there is no way to
determine the IB router from the GID received. Since that is the case
anyway, the invalid bit does not help.

So I am saying is either we say that such setups will have to wait
till IB routing comes up or if it is important suggest a solution.

[The inclusion of LID in the ARP/ND provided the solution but we are
dropping that option].

Vivek