Subject: RE: Why is MTU an issue? (RE: [Ipoverib] Please read - proposed W G termination)

From: Dror Goldenberg <[email-masked]>
Date: Fri, 2 Sep 2005 23:26:05 +0300

> From: Mark Townsley [mailto:[email-masked]]
> Sent: Friday, September 02, 2005 11:12 PM

> Dror Goldenberg wrote:
> 
> >
> >
> > > From: Mark Townsley [mailto:[email-masked]]
> > > Sent: Friday, September 02, 2005 10:43 PM
> >
> > > Dror Goldenberg wrote:
> > >
> > > >
> > > >
> > > > > From: Mark Townsley [mailto:[email-masked]]
> > > > > Sent: Friday, September 02, 2005 4:21 PM
> > > > >
> > > > > Harald Tveit Alvestrand wrote:
> > > > >
> > >
> > > > I think that the right approach is to have a per-neighbor
> > > MTU in the
> > > > L2 interface. And furthermore, MTU can be different between
> > > multicast
> > > > and unicast. We're actually trying to see whether a per
> > > neighbor MTU
> > > > can be added to Linux.
> > > >
> > > On the face of it, "per-neighbor" MTU sounds like a
> > > management nightmare
> > > for anything but a handful of hosts.
> > >
> >
> > You just need to configure for each one of your hosts an MTU for 
> > ipoib-cm, or use the default one. When a connection is 
> made, ipoib-cm 
> > chooses MTU=min(local MTU, remote MTU) and you're done.
> >
> > Maybe you can elaborate more on why you think it's a nightmare ?
> >
> I probably (well, definitely) do not know enough about IB to 
> be making 
> such grand statements. But it seemed to me that if each host 
> may have a 
> different MTU, then all hosts which can talk to one another within a 
> given subnet would have to know the MTU of every other host. So, a 
> classic N^2 problem. Is IB natively point 2 point? Perhaps that would 
> ease my fears.
> 
> - Mark
> 
IB is a fabric based protocol. For each ipoib-cm there is a QP which is
point
to point. When you create this QP and connect it with the peer, you
negotiate
the MTU. Once you're done, you have a QP connected point to point with the
ipoib-cm
stack of your peer. Other peers are connected through other QPs, for which
the 
MTU decision is independent.

The MTU problem is  a n^2 problem but it is solved distributively... Each
host decides which MTU to use for each one of the connections to the peers
it has. In an N hosts fabric, each host may have (N-1) connections to peers.
The MTU selection is trivial, it's the minimum between my local MTU and my
peer MTU, negotiated at the time the connection is established.

_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib

---

From: Mark Townsley <[email-masked]>
Date: Fri, 02 Sep 2005 22:39:41 +0200

Dror Goldenberg wrote:

> IB is a fabric based protocol. For each ipoib-cm there is a QP which 
> is point
> the MTU. Once you're done, you have a QP connected point to point with 
> the ipoib-cm
> stack of your peer. Other peers are connected through other QPs, for 
> which the
> The MTU problem is  a n^2 problem but it is solved distributively... 
> Each host decides which MTU to use for each one of the connections to 
> the peers it has. In an N hosts fabric, each host may have (N-1) 
> connections to peers. The MTU selection is trivial, it's the minimum 
> between my local MTU and my peer MTU, negotiated at the time the 
> connection is established.
>
If you already have a p2p association negotiating and keeping state for 
other things, you are already dealing with a full mesh, so, yes, I see 
that MTU is just a drop in the bucket. Thanks for entertaining my questions.

- Mark

---

From: Michael Krause <[email-masked]>
Date: Fri, 02 Sep 2005 15:09:15 -0700

At 01:26 PM 9/2/2005, Dror Goldenberg wrote:

> > From: Mark Townsley [<mailto:[email-masked]>mailto:[email-masked]]
> > > > From: Mark Townsley 
> [<mailto:[email-masked]>mailto:[email-masked]]
> > > > > > From: Mark Townsley 
> [<mailto:[email-masked]>mailto:[email-masked]]
>IB is a fabric based protocol. For each ipoib-cm there is a QP which is point
>to point. When you create this QP and connect it with the peer, you negotiate
>it has. In an N hosts fabric, each host may have (N-1) connections to 
>peers. The MTU selection is trivial, it's the minimum between my local MTU 
>and my peer MTU, negotiated at the time the connection is established.

It should also be noted that one would not be limited to only one 
connection pair since from a technology perspective, there is no reason why 
multiples cannot be used for QoS purposes / bandwidth management 
purposes.  The same applies to the UD method though few people have 
implemented it as such.

Mike

---

From: Vivek Kashyap <[email-masked]>
Date: Tue, 6 Sep 2005 00:04:06 -0700 (PDT)

snip...

> It should also be noted that one would not be limited to only one connection 
> pair since from a technology perspective, there is no reason why multiples 
> cannot be used for QoS purposes / bandwidth management purposes.  The same 
> applies to the UD method though few people have implemented it as such.

Yes, that is true. One may choose to create multiple connections between two
peers. IPoIB-CM allows this. The implementation can then choose to negotiate 
the same MTU across all connections between the two peers.

Vivek