Subject: [homenet] question: equal-cost multipath?

From: Alia Atlas <[email-masked]>
Date: Tue, 11 Aug 2015 13:47:45 -0400

I am interested to learn what people think about whether equal-cost
multi-path routes are needed in homenet.  Given the previous discussion
about parallel wireless links - which I know I have in my house and can't
use - I've been wondering if these have been considered.

ECMP is critical in the data-center and backbone, but I'm interested in
seeing what the reasoning is as to why it isn't or is needed in the homenet
scenarios.

Thanks,
Alia

P.S. I do expect that any routing protocol can be made to do ECMP, of
course.

---

From: Ole Troan <[email-masked]>
Date: Tue, 11 Aug 2015 19:57:56 +0200

> I am interested to learn what people think about whether equal-cost multi-path routes are needed in homenet.  Given the previous discussion about parallel wireless links - which I know I have in my house and can't use - I've been wondering if these have been considered.
> 
> ECMP is critical in the data-center and backbone, but I'm interested in seeing what the reasoning is as to why it isn't or is needed in the homenet scenarios.

I don’t think homenet has any special requirements with regards to ECMP. I think it has been assumed that will work like in any other network. I certainly expect the homenet to support ECMP.
(parallel wireless links may be a bad example though, as they are probably quite unlikely to be equal.)

the main items of discussion in a homenet context is load balancing across multiple providers. in the IPv6 multi-prefix multi-homing architecture load balancing is done by the hosts, not by the network.

cheers,
Ole

---

From: Toerless Eckert <[email-masked]>
Date: Tue, 11 Aug 2015 11:52:43 -0700

I think the right term  is "multiple WAN paths". There may only be 
a single SP offering useful service and you just need more bandwidth
so you have two links to the SP. 

Any homenet solution where the local address is taken into account
should work as well as it would with different providers. Aka: not
using "normal" ECMP. The only interesting case is when we have a path
selection where only provider side information is used that would be
the same for both links. Eg: Multicast RPF or the like. Thats when
classical ECMP could come into play wrt to WAN links.

The other, case is what i thought to be useful, eg:
in-home powerline adapters with wifi in parallel. And even thought this
mightb e ECMP just by hopcount, it seems useful to consider methods
for non-equal cost load balancing . Not quite sure how much measurement
based babel would support this...

Cheers
    Toerless

On Tue, Aug 11, 2015 at 07:57:56PM +0200, Ole Troan wrote:
> I don???t think homenet has any special requirements with regards to ECMP. I think it has been assumed that will work like in any other network. I certainly expect the homenet to support ECMP.

> _______________________________________________
> homenet mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/homenet

-- 
---
Toerless Eckert, [email-masked]

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Tue, 11 Aug 2015 20:05:53 +0000

RPL enables non-equal cost multipath, Alia. That's the reasonable thing (a MUST if you ask me) to do with wireless connectivity when delivery is statistical and metrics can only provide a limited approximation of transmission chances.

Any DV can do that easily so we should be able to do it with Babel as well. That's called feasible successors in EIGRP.

When things move quickly or conditions change rapidly then you want to trade routing stretch with route quality, which means that in the NECMP routes that you get, it may be that none is shortest path. RPL allows to decide how many and which destinations will benefit from shortest path while others will get stretched routes, and that's the only protocol that can do that on the table and in the whole house. This is how RPL addresses MANET in a really efficient manner.

When devices get critically limited in memory, RPL allows to switch to source routing. Again the only protocol on the table that can do that built in. 

When bandwidth is critically limited (any powerline around?) then the capability to fix routes reactively, that is only when traffic is sent on a broken routing zone, becomes a treat. Built in RPL, nowhere else to be found.

RPL can build different routing instances to reach different gateways or reach then with different optimizations. The former can be done with source/destination routing but the latter is only available with multi topology routing, which may be available in some specific vendors versions of ISIS and, well, is built in RPL.

When only few routes are needed, but could need to be maintained aggressively, RPL has an experimental reactive method akin to AODV call P2P.
Any route at home that needs special treatment like a control loop? 
 
Sadly none of these questions is really on the table for Homenet though they were all requirements for ROLL; fact is we have a std track protocol that can do it all, that is implemented in openWRT and openWSN, and 3 other implementations which run on a few Ks of memory and were tested for interop in Prague.

All the best,

Pascal

> Le 11 août 2015 à 19:47, Alia Atlas <[email-masked]> a écrit :
> 
> P.S. I do expect that any routing protocol can be made to do ECMP, of course.

---

From: Juliusz Chroboczek <[email-masked]>
Date: Tue, 11 Aug 2015 23:02:25 +0200

> RPL enables non-equal cost multipath,

Could you please point out the place in the spec where it is described?

> That's called feasible successors in EIGRP.

Er, no.  Feasible successors are something different.

-- Juliusz

---

From: Juliusz Chroboczek <[email-masked]>
Date: Tue, 11 Aug 2015 23:17:37 +0200

> use - I've been wondering if these have been considered.

As Toerless noted, source-specific routing combined with something like
MP-TCP gives similar effects to ECMP.  Clauz from the Ninux project gave
a talk at Battlemesh with his experiences with such a setup, however, and
the conclusion was that in his particular network he couldn't get it to
work well enough to be useful for his particular application (VPNs to
a central server).

  https://www.youtube.com/watch?v=ztVogLI_ZJs

(That was a great talk, by the way.  We need more talks about what people
were unable to get to work, talks about successes are boring.)

As to ECMP itself -- in a wireless environment, it must be done with care.
If you have two parallel links at the same radio frequency, you will get
collisions between the two subflows, which will get you worse performance
than what you'd get by using a single subflow.  So your routing protocol
needs to be able to distinguish between interfering and non-interfering
routes, and only do ECMP on non-interfering subflows.

In the particular case of Babel, this means that in order to efficiently
do ECMP over wireless you need to use the "Babel-Z" extension [1], which
propagates interference information in order to make better routing
decisions in multi-radio meshes.  I've been refraining from mentioning
this extension on this list, since I've repeatedly been told that meshy
things are out of scope for Homenet.

[1] https://tools.ietf.org/html/draft-chroboczek-babel-diversity-routing
    Note that this extension is not frozen yet, I am planning
    a backwards-incompatible change before I submit it for publication as
    an RFC.  (The TLV format will stay, but the interpretation of some
    values will change; existing implementations will not be impacted.)

Babel does not do ECMP yet (it does do feasible successors, of course).
If there's interest, and somebody willing to do some testing for me
(empirical data only, please), then I think I could conjure something up
pretty quickly.

-- Juliusz

---

From: Michael Richardson <[email-masked]>
Date: Tue, 11 Aug 2015 17:53:52 -0400

I don't think that ECMP is useful/interesting *within* the Homenet.

It is certainly true that having two DSL links "bonded" is regularly done
(usually using MPPP), but that presents as a single link.  Some will want two
CPE routers for reasons of redundancy on their multi-path uplink. My opinion
is that we are rapidly getting to "professional managed" networks here, even
if the professional the ISP or the very sophisticated home-based IT worker.

Being able to identify backup links and rapidly swing traffic is important,
but I think that all the protocols can do that.

I think that MPTCP is more likely to bring significant benefit to
applications that need to leverage multiple ISPs.

--
Michael Richardson <[email-masked]>, Sandelman Software Works
 -= IPv6 IoT consulting =-

---

From: "Acee Lindem (acee)" <[email-masked]>
Date: Tue, 11 Aug 2015 23:06:17 +0000

On 8/11/15, 5:17 PM, "homenet on behalf of Juliusz Chroboczek"
<[email-masked] on behalf of [email-masked]>
wrote:

>MP-TCP gives similar effects to ECMP.

One of the requirements of homenet is that you don’t modify the hosts so
MTCP is not an alternative to ECMP.

Thanks,
Acee 

> Clauz from the Ninux project gave

---

From: Alia Atlas <[email-masked]>
Date: Tue, 11 Aug 2015 19:25:13 -0400

Hi Michael, Juliusz, & Pascal,

Thanks for your thoughts.   I understand about the different upstream
providers.  However,  inside the home, if there are multiple paths, I can
also picture it being useful to use them (backups to a NAS, multiple video
streams, etc).

Whether they are simply equal-cost or unequal but safe to use paths is an
interesting detail,  but I think the main point is whether parallel paths
will be supported.

The concerns about wifi links interfering with each other is interesting.
I wonder if that is always a local decision for one end of the links or
whether a link from A to B and one from C to D would need to be
coordinated?  I'm tempted to want a nice abstraction layer, but I also
sense that it isn't quite that simple.

It sounds to me like using multiple paths (ECMP or otherwise) is something
that hasn't been clearly nailed down in the requirements?

Regards,
Alia

---

From: Juliusz Chroboczek <[email-masked]>
Date: Wed, 12 Aug 2015 01:26:22 +0200

> One of the requirements of homenet is that you don’t modify the hosts

More precisely, Homenet must not *require* host modifications.  Having
features that yield better behaviour or performance for modified hosts is
hopefully allowed -- otherwise, we might as well give up on source-specific
routing.

(Acee, when did you last upgrade the OS on your laptop?  On your mobile
phone?  And when did you last upgrade the firmware on the $50 wifi router
you bought when your daughter complained about poor wifi coverage?  Home
networks are different from enterprise networks, like it or not.)

-- Juliusz

---

From: Juliusz Chroboczek <[email-masked]>
Date: Wed, 12 Aug 2015 01:30:06 +0200

> sense that it isn't quite that simple. 

That's why I require Babel-Z.  Babel-Z attaches enough information to each
route so that the decision becomes local to the router performing ECMP.

-- Juliusz

---

From: Lorenzo Colitti <[email-masked]>
Date: Wed, 12 Aug 2015 09:47:47 +0900

> seeing what the reasoning is as to why it isn't or is needed in the homenet

ECMP is critical in the datacenter and backbone because those networks are
designed to provide the E ("equal") in ECMP. Because the links are equal,
it's easy to load-balance over them without needing to do complicated stuff
like traffic engineering - you just treat an N-way ECMP bundle as a link N
times bigger, and hash across it. That does not happen in home networks,
which are more grown than designed.

Having a homenet load-balance Internet-bound across multiple provides is a
non-starter because it is presumed that said providers will employ BCP38
filtering. It's possible for the *hosts* to load-balance across different
providers by simply sending their packets with different source addresses
(and, in some cases, different routers).

---

From: [email-masked]
Date: Tue, 11 Aug 2015 18:53:18 -0700

> On Aug 11, 2015, at 5:47 PM, Lorenzo Colitti <[email-masked]> wrote:
> 
> Having a homenet load-balance Internet-bound across multiple provides is a non-starter because it is presumed that said providers will employ BCP38 filtering. It's possible for the *hosts* to load-balance across different providers by simply sending their packets with different source addresses (and, in some cases, different routers).

This is why you need an overlay. I do this at my house at the moment and I have a single address for each of my hosts. 

Dino

---

From: Lorenzo Colitti <[email-masked]>
Date: Wed, 12 Aug 2015 11:03:28 +0900

On Wed, Aug 12, 2015 at 10:53 AM, <[email-masked]> wrote:

> > Having a homenet load-balance Internet-bound across multiple provides is
> a non-starter because it is presumed that said providers will employ BCP38
> This is why you need an overlay. I do this at my house at the moment and I
> have a single address for each of my hosts.
>

Ok, but that forces you to tunnel all traffic through an out-of-home tunnel
endpoint point, which is sort of a non-starter for normal residential
users. I mean - which ISP is going to provide such tunnel endpoints just to
make it easier for their customers to *also* be customers of their
competitors? How many users are going to want to pay a third party to
provide that endpoint to them, when they're already paying for an Internet
connection?

---

From: [email-masked]
Date: Tue, 11 Aug 2015 19:23:18 -0700

> Ok, but that forces you to tunnel all traffic through an out-of-home tunnel endpoint point, which is sort of a non-starter for

This would be only external traffic. For traffic inside of the home you use an IGP with host route support so one address can move across subnets inside the home. 

> normal residential users. I mean - which ISP is going to provide such tunnel

You don't necessarily tunnel to the ISP. You CAN tunnel to the ISP if it gives you new services you would like to buy. Otherwise, you tunnel to other sites directly. 

> endpoints just to make it easier for their customers to *also* be customers of their competitors? How many users are going to 

It doesn't work in that direction. An ISP will tunnel to a partner (notice I didn't say competitor) customer to give new services without the partner needing to be involved. The partner CAN be involved but not required. 

> want to pay a third party to provide that endpoint to them, when they're already paying for an Internet connection?

You and I tunnel to each other. We pay no one and we talk to each other without any coordination from anyone else. If I want some DoS mitigation or service chaining, I pay my ISP to do it. But you pay nothing. 

Dino

---

From: Lorenzo Colitti <[email-masked]>
Date: Wed, 12 Aug 2015 11:24:17 +0900

On Wed, Aug 12, 2015 at 11:23 AM, <[email-masked]> wrote:

> You and I tunnel to each other. We pay no one and we talk to each other
> without any coordination from anyone else. If I want some DoS mitigation or
> service chaining, I pay my ISP to do it. But you pay nothing.
>

Ah, you mean "use LISP"? Got it.

---

From: [email-masked]
Date: Tue, 11 Aug 2015 19:39:56 -0700

Probably doesn't come as a surprise. :-)

Dino

> On Aug 11, 2015, at 7:24 PM, Lorenzo Colitti <[email-masked]> wrote:
>

---

From: Alia Atlas <[email-masked]>
Date: Tue, 11 Aug 2015 22:54:07 -0400

Hi Lorenzo,

> which are more grown than designed.

ECMP applies beyond link bundles.  Of course, equal-cost can be hard to do
- and one can safely use downstream paths.  The relevant question is
whether traffic is expected to be able to take multiple paths to allow
load-balancing.

> (and, in some cases, different routers).

Yes, of course if one is doing src-dest routing, the multiple paths would
have to be valid for the route.

Regards,
Alia

---

From: Lorenzo Colitti <[email-masked]>
Date: Wed, 12 Aug 2015 12:16:33 +0900

> whether traffic is expected to be able to take multiple paths to allow

Yes, ECMP applies beyond bundles, but unless your metrics are very
simplistic, you won't actually encounter equal-cost paths unless you design
the network with ECMP in mind. So you'd need non-equal cost multipath for
multipath to be useful in a homenet.

---

From: Alia Atlas <[email-masked]>
Date: Tue, 11 Aug 2015 23:20:55 -0400

Hi Lorenzo,

On Tue, Aug 11, 2015 at 11:16 PM, Lorenzo Colitti <[email-masked]>
wrote:

>> ECMP is critical in the datacenter and backbone because those networks
>>> are designed to provide the E ("equal") in ECMP. Because the links are
>>> equal, it's easy to load-balance over them without needing to do
>>> complicated stuff like traffic engineering - you just treat an N-way ECMP
>>> bundle as a link N times bigger, and hash across it. That does not happen
>>> in home networks, which are more grown than designed.
>>>
>>
>> ECMP applies beyond link bundles.  Of course, equal-cost can be hard to
>> do - and one can safely use downstream paths.  The relevant question is

Yes - downstream paths, as I already said.  That is going to next-hops that
are closer to the destination than the computing router.  As long as your
next-hop's distance to the destination is strictly decreasing, it is safe
to use.

There are two questions.  First, is the desirable to load-balance among
different paths useful/necessary/unnecessary in homenet?  Second, is that
accomplished with metric assignment that encourages equal-cost, are
downstream paths used, and/or is there a way of doing explicit paths?

Regards,
Alia

---

From: Henning Rogge <[email-masked]>
Date: Wed, 12 Aug 2015 07:00:16 +0200

On Wed, Aug 12, 2015 at 5:20 AM, Alia Atlas <[email-masked]> wrote:
> next-hop's distance to the destination is strictly decreasing, it is safe to
> use.

In theory yes... in practice (especially in the presence of wireless
links) you might just spam your airspace by producing lots of
additional collisions.

> downstream paths used, and/or is there a way of doing explicit paths?

Making the metric too simplistic means you cannot deal with a
heterogeneous network like a mixture of ethernet (of different link
speeds) and wifi (with LOTS of different link speeds and frame loss).

Henning Rogge

---

From: Mikael Abrahamsson <[email-masked]>
Date: Wed, 12 Aug 2015 07:28:22 +0200 (CEST)

On Tue, 11 Aug 2015, Alia Atlas wrote:

> different paths useful/necessary/unnecessary in homenet?  Second, is 
> that accomplished with metric assignment that encourages equal-cost, are 

I'd say it's useful but not necessary. It's hard to justify to put it in 
as a requirement but if the routing protocol would provide it as-is, I'd 
definitely wouldn't want to rip it out. If the user happens to connect two 
routers with two different cables, why not use the capacity available? 
This would of course still mean you look at media type and speed when 
setting metrics so that you don't send some traffic over a slow and 
unreliable link if a fast and reliable link is available.

Doing explicit paths sounds like TE? No, that's never been discussed and I 
don't think anyone is envisioning this happening because homenet has so 
far not been about overlay networks.

If we were to rev the architecture document, I'm sure it would look 
different than what it looks like right now. There are things that weren't 
brought up in it that people just thought were obvious (I believe).

-- 
Mikael Abrahamsson    email: [email-masked]

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Wed, 12 Aug 2015 09:41:48 +0000

Argll.

We manage to disagree on definitions? There's multipath and there's load balancing. Multipath is a topological thing, load balancing is one thing you can do with that topology. Maybe we disagree on definition because you are grouping the two together?

So if we have to define (N)ECMP, here's my take:
- Shortest path routing creates DODAGs to a destination D wherein all paths to get to D have an equal cost. When a network is not engineered, this usually means forming trees. 
- Greedy NECMP forms DODAGs whereby different paths may have different costs; the resulting topology still exhibit single points of failure (e.g. the node closest to destination does not have a plan B). 
- Non-greedy NECMP routing such as ARCs allow to roll back and reroute around the block. 

Multipath is important inside the home when the links are not reliable; that's not for complex load balancing stuff but for the purpose of per-packet reroute. If a wireless transmission fails between A and B, the statistical chances of successful transmission drop with the number of retries. It makes sense to add diversity to avoid the cause of the failure, and spatial diversity using multipath is one way to achieve this, which is particularly efficient to combat multipath fading. On wireless, equal cost is a vague approximation anyway so NECMP is perfectly acceptable and, in fact, desirable.

With the definition above, feasible successors used adequately can bring useful NECMP at home.

Unless I really missed something, RPL and BABEL have the same feasibility condition, which your Babel RFC explains extremely well, and that inherits from a long line of successful DV protocols such as EIGRP, and can be traced back to JJ's DUAL work. You'll find a very similar logic in LFA (section 3-2 of RFC5286). RPL's rules are in section 8.2.2.4 of RFC 6550.

The thing BABEL appears to be missing is this: 

- to really benefit from NECMP, (this is really driven by OF policy but the general rule is that) RPL builds a parent set as opposed to selecting a shortest path next hop. In practice the parent set is actively used to forward retries. Being a routing protocol we did not really discuss the forwarding in the spec, but the behavior is that the packet for which L2 transmission failed is passed back to L3 to be eventually retried on another successor.

- On top of that, RPL has an optional tweak that a node may increase its Rank (up to a bounded limit) in the hope of getting more parents and/or retain connectivity with the accepted risk of Count-To-Infinity and data-path micro-loops. Because packets are tagged with the Rank of the sender and the direction (increasing or decreasing Rank), a packet along a micro-loop is detected and eliminated immediately.

Cheers,

Pascal

> -----Original Message-----
> From: Juliusz Chroboczek [mailto:[email-masked]]
> Sent: mardi 11 août 2015 23:02
> To: Pascal Thubert (pthubert) <[email-masked]>
> Cc: Alia Atlas <[email-masked]>; HOMENET <[email-masked]>
> Subject: Re: [homenet] question: equal-cost multipath?
>

---

From: Teco Boot <[email-masked]>
Date: Wed, 12 Aug 2015 11:53:30 +0200

Alia,

I still think the Homenet WG should pay far more attention to seamless WiFi handover, it is critical. I did not wait for an outcome, I have my WLC :-).

On host based multi-path, there is a highly experimental MPTCP based solution, that could work better than my WLC infra. And cheaper... It uses tricks with One NIC + multiple STA + (optional) channel switch + sleep mode.
https://www.ietf.org/proceedings/89/slides/slides-89-mptcp-5.pdf

Teco

> Op 11 aug. 2015, om 19:47 heeft Alia Atlas <[email-masked]> het volgende geschreven:
>

---

From: Juliusz Chroboczek <[email-masked]>
Date: Wed, 12 Aug 2015 11:57:37 +0200

> unless your metrics are very simplistic, you won't actually encounter
> equal-cost paths

Listen to the man.

But even if you happen to have equal cost paths, or design a suitable
heuristic for non-equal-cost multipath, it's not clear to me that ECMP is
beneficial.  ECMP might increase throughput, granted, but it is likely to
increase latency and packet loss.  I don't see many cases of intra-home
traffic that are likely to be limited by throughput, except backups.  And
who does backups in any case?

So I'd be very cautious about deploying ECMP without a clear understanding
of its performance implications in the particular case of the home -- does
it bring perceptible benefits to the users, which, at the end of the day,
is the only thing that really counts.  Which should of course not prevent
us from experimenting, quite the opposite.

-- Juliusz

---

From: Juliusz Chroboczek <[email-masked]>
Date: Wed, 12 Aug 2015 12:27:54 +0200

> I still think the Homenet WG should pay far more attention to seamless
> WiFi handover,

We're currently working on that with the WLAN-SI guys.  They've got
a nationwide mesh network (it's a small country, granted, and they use the
public Internet infrastructure whenever possible), and while they're not
interested in roaming at a global scale, they insist on seamless roaming
within a small cluster of routers (home or village sized), and they insist
on no host changes and good support for Android hosts.

We'll write to this list when we know what works.

In case it doesn't show, I'm enthusiastic about the energy and competence
of the WLAN-SI core team.  They make a serious effort in translating
everything they do into English, and they've got a nice node database
online (even nicer one ready, but not deployed yet):

  https://wlan-si.net/
  https://nodes.wlan-si.net/

-- Juliusz

---

From: Mikael Abrahamsson <[email-masked]>
Date: Wed, 12 Aug 2015 13:14:40 +0200 (CEST)

> increase latency and packet loss.  I don't see many cases of intra-home

Why would it increase latency and packet loss (generally)? If I put 2 
separate wired connections between two homenet routers, I would just like 
it to use both if they seem to work.

> So I'd be very cautious about deploying ECMP without a clear 
> understanding of its performance implications in the particular case of 
> the home -- does it bring perceptible benefits to the users, which, at 
> the end of the day, is the only thing that really counts.  Which should 
> of course not prevent us from experimenting, quite the opposite.

I wouldn't want to do ECMP across widely different mediums and make it 
super advanced with unequal load balancing etc, but if I hook up two wired 
GIGE between two homenet routers, why not use it?

---

From: Teco Boot <[email-masked]>
Date: Wed, 12 Aug 2015 13:19:19 +0200

> Op 12 aug. 2015, om 12:27 heeft Juliusz Chroboczek <[email-masked]> het volgende geschreven:
> 

Great!

> We'll write to this list when we know what works.

Boring. I'm interested in what doesn't work :-)

While I pay for it, I never use the millions of WiFi access points I can use here in the Netherlands. I tried it once, walking in a small city. At the time the handover was completed, the connectivity was gone. It might have to do with IEEE and IETF mismatch. Same SSID shall have same IP subnet (IEEE) versus each link has its own subnet (some of IETF, no formal statement...).

Teco

---

From: Ray Bellis <[email-masked]>
Date: Wed, 12 Aug 2015 12:30:56 +0100

On 12 Aug 2015, at 00:26, Juliusz Chroboczek <[email-masked]> wrote:
> hopefully allowed -...

yes, that's correct.

ray

P.s I'm on vacation now for two weeks with limited net access. Responses may be delayed and/or brief.

---

From: Juliusz Chroboczek <[email-masked]>
Date: Wed, 12 Aug 2015 13:41:26 +0200

Removed Alia from CC, I think she's on the list.

> We manage to disagree on definitions? There's multipath and there's load
> balancing. Multipath is a topological thing, load balancing is one thing
> you can do with that topology. Maybe we disagree on definition because
> you are grouping the two together?

Yes, that's right.  I'm pretty sure that Alia meant "ECMP load sharing"
when she wrote "ECMP" in her initial mail.  This would appear to be
consistent with Cisco usage.

Merely keeping a redundant routing table is called "feasible successors"
in the Cisco litterature, and "redundant feasible routes" in Babel. I suggest
that we avoid calling this multipath, in order to avoid confusion with
ECMP and MP-TCP.

> Unless I really missed something, RPL and BABEL have the same
> feasibility condition, which your Babel RFC explains extremely well, and
> that inherits from a long line of successful DV protocols such as EIGRP,
> and can be traced back to JJ's DUAL work.

Agreed, but I don't think that's what Alia was speaking about.

> The thing BABEL appears to be missing is this [...]

Right in both cases.  Babel's starvation avoidance and RPL's greediness
(how come both are food metaphors?  because we both live in France?)
represent different tradeoffs, which reflects the different design
criteria of the protocols -- Babel tries to be safe, L2-agnostic and easy
to implement, while RPL is very serious about low power and extremely low
resources, at the cost of unsafe tweaks and vastly increased implementation
complexity.

-- Juliusz

---

From: Juliusz Chroboczek <[email-masked]>
Date: Wed, 12 Aug 2015 14:05:26 +0200

> Why would it increase latency and packet loss (generally)? If I put
> 2 separate wired connections between two homenet routers, I would just
> like it to use both if they seem to work.

I think Alia was speaking about parallel WiFi links.  That's pretty
common -- double-band routers are slowly becoming the norm.

If you load-share between the two links, the resulting latency of the
stream will be the largest of the two latencies (assuming in-order
delivery).  The packet loss will be the average, which is larger than the
packet loss of the link with the lower packet loss of the two.  Plus
there's the interference issues.

I'm not saying it cannot be done, I'm just saying that we mustn't do it
until we fully understand the tradeoffs.

> if I hook up two wired GIGE between two homenet routers, why not use it?

Agreed.  But even if you have a NAS that can handle more than a Gbit/s
sustained, this use case is rather marginal.

-- Juliusz

---

From: Michael Richardson <[email-masked]>
Date: Wed, 12 Aug 2015 08:15:37 -0400

Alia Atlas <[email-masked]> wrote:
> providers. However, inside the home, if there are multiple paths, I
> can also picture it being useful to use them (backups to a NAS,
> multiple video streams, etc).

I find it difficult to construct a multi-path situation where the paths
are equal (i.e. not wifi vs GbE), and where there are actually more than
one path the entire way.

Can you construct such a network out of laptops, desktops, home routers
and NAS?  Each have one GbE port and a WIFI interface?   

> The concerns about wifi links interfering with each other is
> interesting. I wonder if that is always a local decision for one end
> of the links or whether a link from A to B and one from C to D would
> need to be coordinated? I'm tempted to want a nice abstraction layer,
> but I also sense that it isn't quite that simple.

Wifi with an AP is a time-division star topology at layer-2, so the
AP decides who gets bandwidth...

---

From: Mikael Abrahamsson <[email-masked]>
Date: Wed, 12 Aug 2015 14:17:02 +0200 (CEST)

> common -- double-band routers are slowly becoming the norm.

I don't think she was speaking for wifi links, I think she was speaking of 
any kind of link.

> delivery).  The packet loss will be the average, which is larger than 
> the packet loss of the link with the lower packet loss of the two. 
> Plus there's the interference issues.

When you say ECMP to a routing person, it's generally something that is 
set up by an IGP or using BGP. It's something that has the same metric, 
and the FIB then starts load sharing over these (equal IGP cost) links. In 
core or access networks networks this is typically done by hashing 5 tuple 
information so each flow gets in-order delivery, but if you have multiple 
flows (which you usually have on these kinds of links, many tens of 
thousands of them), you can use most of the available capacity.

I believe this is what Alia was refering to.

> sustained, this use case is rather marginal.

Well, if you have a NAS that supports two NICs, you can for instance use 
LACP to bond them together and then when you have multiple clients, the 
router can hash each L4 session to (hopefully) a separate link in the 
homenet if they're several routers away.

So as I said before: it's desireable for me that the homenet routing 
protocol supports ECMP and can take use parallel links between devices in 
simultaneous use.

2 or 4 GE links is going to be cheaper than a single 10GE link for quite 
some time. While some might believe this is pure SciFi that a home would 
need this kind of capacity, I'd say in 5-10 years (which is when I think 
homenet really should have taken off), this is going to be a lot more 
common. So ruling out ECMP right now seems short sighted.

---

From: Mikael Abrahamsson <[email-masked]>
Date: Wed, 12 Aug 2015 14:20:07 +0200 (CEST)

> and NAS?  Each have one GbE port and a WIFI interface?

There are already announced services that are more than 1GE:

http://www.pcworld.com/article/2947712/networking-hardware/comcasts-2-gigabit-service-will-cost-300-plus-1000-for-installation-and-activation.html

So if this is now the case, why would I not want to be able to put 2x1GE 
between my homenet routers to actually make use of this bandwidth?

---

From: "Eric Vyncke (evyncke)" <[email-masked]>
Date: Wed, 12 Aug 2015 12:20:29 +0000

>
>
>
>While I pay for it, I never use the millions of WiFi access points I can
>use here in the Netherlands. I tried it once, walking in a small city. At
>the time the handover was completed, the connectivity was gone. It might
>have to do with IEEE and IETF mismatch. Same SSID shall have same IP
>subnet (IEEE) versus each link has its own subnet (some of IETF, no
>formal statement...).

Of course, if every SSID has its own 192.168.0.0/24, oups, this is legacy
IP, so, not a Homenet topic :-O

Sorry could not resist

-éric

---

From: Juliusz Chroboczek <[email-masked]>
Date: Wed, 12 Aug 2015 14:50:32 +0200

> In core or access networks networks this is typically done by hashing
> 5 tuple information so each flow gets in-order delivery, but if you have
> multiple flows (which you usually have on these kinds of links, many
> tens of thousands of them), you can use most of the available capacity.

[...]

> protocol supports ECMP and can take use parallel links between devices in

Could you please spell out a Homenet-related use case for that?  Something
that I can explain to my niece?  (My niece is smart, interested in technology,
and has a lot of patience for good explanations.  Yeah, I like her a lot.)

-- Juliusz

---

From: Mikael Abrahamsson <[email-masked]>
Date: Wed, 12 Aug 2015 15:06:03 +0200 (CEST)

On Wed, 12 Aug 2015, Juliusz Chroboczek wrote:

> Could you please spell out a Homenet-related use case for that? 
> Something that I can explain to my niece?  (My niece is smart, 
> interested in technology, and has a lot of patience for good 
> explanations.  Yeah, I like her a lot.)

ISP+--+
       A+-+NAS
       +
       |
       +
   C+-+B+-+D

It would be good if A=B could be two parallel GE so that D could access 
the ISP at gig speed at the same time C accesses the NAS at gig speed.

Yes, this relies on a bit of luck with the hashing of the sessions, or 
that one has multiple sessions (torrent for instance), but still...

---

From: Teco Boot <[email-masked]>
Date: Wed, 12 Aug 2015 15:18:33 +0200

> Op 12 aug. 2015, om 14:20 heeft Eric Vyncke (evyncke) <[email-masked]> het volgende geschreven:
> 
>> 
>> 
>> 

I don't know what they used before. Tried again, now I have 10.32/12. No IPv6. Lots of multicast and broadcast suppression, if not all.
I give it another shot, maybe roaming is better these days.

I upgraded my 3-play subscription. I hoped for IPv6. Not. They stopped roll-out because too much helpdesk calls.

This ISP guest network is a Homenet topic. I've got a new STB, with ethernet switch and WiFi AP. Next version would have MoCa. How can this provider use the homenet to extend its guest network? If it can't, why would this ISP be interested in homenet?

How works *seamless* WiFi handover with same SSID, with different IPv6 subnets?
Single subnet for a neighborhood, with all the multicast suppression? Single subnet for a country?

Teco

---

From: Mikael Abrahamsson <[email-masked]>
Date: Wed, 12 Aug 2015 15:23:04 +0200 (CEST)

> How works *seamless* WiFi handover with same SSID, with different IPv6 subnets?

It doesn't work, regardless if IPv4 or IPv6 is used.

We need to move away from hosts seeing the address as permanent. SHIM6 
seemed like a decent idea, but didn't pan out for several reasons. MPTCP 
will have a rough time as well, we'll see how that will work out.

I myself believe in "temporal overlap", ie be connected to both networks 
for some time, finish whatever you were doing (if you were downloading 
video snippets for instance) and then start using the new connction. A lot 
of short-lived traffic could be handled like that. However, this wouldn't 
solve real time communictation, there the protocol itself would need to 
tell other end to start using another address for continued communication.

---

From: Juliusz Chroboczek <[email-masked]>
Date: Wed, 12 Aug 2015 15:27:34 +0200

> the ISP at gig speed at the same time C accesses the NAS at gig speed.

Are you assuming that my niece has gigabit to the ISP and needs to
transfer terabytes of data over the Internet while simultaneously
backing up her movie collection at gigabit speed?

Mikael, you appear to be feature-driven -- to quote somebody I like, 
"moar TLVs, better protocol!".  Let's please step back, and think about
what the Homenet user actually needs or wants.

-- Juliusz

---

From: Teco Boot <[email-masked]>
Date: Wed, 12 Aug 2015 15:34:49 +0200

> Op 12 aug. 2015, om 15:23 heeft Mikael Abrahamsson <[email-masked]> het volgende geschreven:
> 

Right.

> 
> We need to move away from hosts seeing the address as permanent. SHIM6 seemed like a decent idea, but didn't pan out for several reasons. MPTCP will have a rough time as well, we'll see how that will work out.

Nice long-term goal. 

> 
> I myself believe in "temporal overlap", ie be connected to both networks for some time, finish whatever you were doing (if you were downloading video snippets for instance) and then start using the new connction. A lot of short-lived traffic could be handled like that. However, this wouldn't solve real time communictation, there the protocol itself would need to tell other end to start using another address for continued communication.

IEEE802.11 spec is otherwise. Unless dial NIC or tricks with sleep mode, a station is connected to a single AP. There is no "temporal overlap". And what is temporal? For me, it is walking thru my house with active voice call. Could be over an hour.

Teco

---

From: Mikael Abrahamsson <[email-masked]>
Date: Wed, 12 Aug 2015 15:37:26 +0200 (CEST)

> what the Homenet user actually needs or wants.

I guess we're not even close (again) on agreeing what a homenet is and 
what the requirements are.

I live in a country where gigabit ethernet connectivity to households is 
not at all uncommon (today). Sometimes it's included in the rent, 
sometimes you have to pay 10-20EUR extra to upgrade from 100/100 to 
gigabit. Sometimes it costs 50-80EUR per month to get this service (then 
people usually settle for 100/100 or something like that that might cost 
30-40EUR per month. It all depends on what service provider is available 
where you live.

I generally like my home network to be of higher speed than my Internet 
access line. I would imagine this is what most people actually want and 
need, at least that's what the ISP would prefer to people can actually get 
speeds close to what they're buying.

Do you know how many ISPs get complaints when customers cannot get more 
than 200-400 megabit/s of transfer speed on their gig connections, because 
their home equipment sucks? Lots of them.

So I want the homenet to work all the time, and be fast. ECMP could be one 
component in this going forward. Why are you so opposed to his, I thought 
ECMP was available in (basically) all routing protocols?

---

From: "Eric Vyncke (evyncke)" <[email-masked]>
Date: Wed, 12 Aug 2015 13:39:54 +0000

>Single subnet for a country?

Not sure whether 'community wifi' belongs to this alias but anyway, there
are working solutions with host routes and/or tunnels... Nothing optimum
and probably not really scalable for a whole country.

I will take your single /64 for the whole country as a joke :-)  OTOH,
there are actually solutions for this kind of deployment, let's wait until
Pascal chimes in ;-)

-éric

---

From: Mikael Abrahamsson <[email-masked]>
Date: Wed, 12 Aug 2015 15:41:15 +0200 (CEST)

On Wed, 12 Aug 2015, Teco Boot wrote:

> IEEE802.11 spec is otherwise. Unless dial NIC or tricks with sleep mode, 
> a station is connected to a single AP. There is no "temporal overlap". 
> And what is temporal? For me, it is walking thru my house with active 
> voice call. Could be over an hour.

Yes, so your wifi device needs to support being connected to two APs for a 
short duration of time (probably a few seconds), so your new address can 
be communicated to whoever your voip call is with, so the session can be 
moved over to that new address.

I know this isn't available currently, but I want to move from "it doesn't 
work" to "how can we make this work without having lots of tunnels all the 
time".

"mosh" is a good example of an application that does what I want.

I come with my laptop with an ongoing mosh/ssh session, I plug in my 
laptop, I see it connect to the wired network, 3 seconds later I disable 
my wifi, and my mosh session still works. I wish all protocols worked like 
this.

---

From: Teco Boot <[email-masked]>
Date: Wed, 12 Aug 2015 15:44:27 +0200

My niece can't wait for getting her first NBASE-T gear. She tells me this is much cheaper and faster than a couple of UTP cables, where she also cannot utilize the bundled throughput for her single connection!!!

Teco

> Op 12 aug. 2015, om 15:27 heeft Juliusz Chroboczek <[email-masked]> het volgende geschreven:
>

---

From: Juliusz Chroboczek <[email-masked]>
Date: Wed, 12 Aug 2015 15:46:30 +0200

> ECMP could be one component in this going forward. Why are you so
> opposed to [t]his

I'm not opposed to ECMP, quite the opposite, I think it would be a fun
thing to implement.  I'm trying to understand the use cases, since I have
the policy of not implementing anything until it is requested by my users
(fewer TLVs = better protocol).

Alia's use case is different from yours, and the two may require different
heuristics (Alia's: balance over any two non-interfering links,
yours: balance over wired links only).  So finding out what the realistic
use cases are is necessary in order to get something done.

-- Juliusz

---

From: Juliusz Chroboczek <[email-masked]>
Date: Wed, 12 Aug 2015 16:06:10 +0200

> My niece can't wait for getting her first NBASE-T gear.

You got my interest here.  What is she doing to find 1Gbit/s limiting?

(But my niece is still cooler than your niece.)

-- Juliusz

---

From: Ray Hunter <[email-masked]>
Date: Wed, 12 Aug 2015 16:23:09 +0200

Alia Atlas wrote:
> multi-path routes are needed in homenet.  Given the previous 
> discussion about parallel wireless links - which I know I have in my 
> house and can't use - I've been wondering if these have been considered.
>
> ECMP is critical in the data-center and backbone, but I'm interested 
> in seeing what the reasoning is as to why it isn't or is needed in the 
> homenet scenarios.
>
> Thanks,
> Alia
>

I was surprised ECMP did NOT work when I tested the latest homenet code. 
All the elements are capable of doing it, but it wasn't implemented.

One reason IMHO that it's potentially useful (but not essential) is that 
(most) plastic homenet routers include a L2 switch in front of the L3 
interface(s). Therefore the L2 interface doesn't actually physically go 
down when cables are unplugged/ links are unstable. Without ECMP you get 
many seconds of complete black holing whilst the router figures out at 
L3 (RP hellos or HNCP) that the interface is actually down and the 
neighbour has gone away. With ECMP you just get bad packet loss if load 
balancing is per packet.

---

From: Teco Boot <[email-masked]>
Date: Wed, 12 Aug 2015 16:27:05 +0200

> Op 12 aug. 2015, om 16:06 heeft Juliusz Chroboczek <[email-masked]> het volgende geschreven:
> 

She can't make it to One Direction. She prepares for V-concert, with her VR-goggles.

Back to serious. More bandwidth is still important, AP's will be connected via multi-Gig links. But I see a shift towards low latency, ubiquity, reliability etc. (e.g. check the 5G requirements).

Teco

---

From: Mikael Abrahamsson <[email-masked]>
Date: Wed, 12 Aug 2015 16:31:24 +0200 (CEST)

> (fewer TLVs = better protocol).

I've been involved in implementing IPv6 for almost 10 years now. If I 
waited until it became a user requirement I would still have done nothing.

People want "working Internet" and they want us to figure out how to make 
it work for them without them having to be bothered to actually figure out 
how things work (too much). If they cobble together their homenet with 
some cables and someone tells them to put two cables between two 
equipment, I'd like to see the capacity used.

Do you even need more TLVs? I'm a bit rusty on DV since I've mostly been 
exposed to OSPF/ISIS and BGP all my life, but all of these can be made to 
load-share as soon as they discover two paths with identical metrics.

---

From: Mikael Abrahamsson <[email-masked]>
Date: Wed, 12 Aug 2015 16:35:13 +0200 (CEST)

On Wed, 12 Aug 2015, Ray Hunter wrote:

> gone away. With ECMP you just get bad packet loss if load balancing is 
> per packet.

Ugh, per-packet load balancing is bad, mmkay. It causes frequent packet 
reordering and we don't want that.

So I don't believe at all in ECMP "solving" that problem.

Also, a future homenet router, I would imagine it's going to look 
different from what current home gateways look like, it's going to have 
multiple "wan" ports that actually will have direct access to the SOC 
without any L2 chip in between, or at least the L2 switch is going to have 
a lot more capacity towards the SOC than 1-2 gigs. Also, there is going to 
have to be integration between the switch and the OS so the OS can detect 
when the switch port goes down and not just wait until the keepalives time 
out.

---

From: Ray Hunter <[email-masked]>
Date: Wed, 12 Aug 2015 16:35:48 +0200

Teco Boot wrote:
>> Op 12 aug. 2015, om 12:27 heeft Juliusz Chroboczek<[email-masked]>  het volgende geschreven:
>>
+1
> While I pay for it, I never use the millions of WiFi access points I can use here in the Netherlands. I tried it once, walking in a small city. At the time the handover was completed, the connectiv…
I second this end-user experience: 2 million access points in one of the 
most densely populated areas on Earth, very strong signal coverage, but 
a pretty useless experience because handoff is so poor.

End host modifications aren't in Homenet scope, but wifi roaming is 
something we really need to address (to at least clarify what Homenet 
will/ will not support in terms of SSID etc. )

---

From: Juliusz Chroboczek <[email-masked]>
Date: Wed, 12 Aug 2015 16:47:45 +0200

> exposed to OSPF/ISIS and BGP all my life,

I don't know, I need to think it over.  Right now, I can see the following
issues:

  - loop avoidance: BGP has the AS_SET attribute for that, do we need
    something similar in Babel, or can we get away with the current
    feasibility signalling?

  - interference avoidance: do we want to balance over any two paths (in
    which case no signalling is needed), over non-interfering paths (in
    which case the Babel-Z extension provides all you need), over any
    wired paths (again, Babel-Z is enough), or over fully disjoint and
    non-interfering paths (in which case I'll need to extend Babel-Z in an
    incompatible manner or design a new extension)?

> but all of these can be made to load-share as soon as they discover two
> paths with identical metrics.

Yeah, but if you do it naively, performance will get worse, not better.
(The IS-IS community has it easy, they just implement things and don't
care if they make things worse.)

Folks, if anyone has done a biblio search for ECMP, I'll be grateful for
pointers.  If not, I'll take an afternoon of my copious free time and do
it myself.  (But don't hold your breath, I've got my hands full with
analysing the Battlemesh results and getting shncpd into shape, and I have
a serious teaching load that starts in early September).

-- Juliusz

---

From: Michael Richardson <[email-masked]>
Date: Wed, 12 Aug 2015 11:21:11 -0400

Mikael Abrahamsson <[email-masked]> wrote:
    >> Can you construct such a network out of laptops, desktops, home
    >> routers and NAS?  Each have one GbE port and a WIFI interface?

    > So if this is now the case, why would I not want to be able to put
    > 2x1GE between my homenet routers to actually make use of this
    > bandwidth?

Today, I would manually bond (LACP) those two interfaces at layer-2, which
would result in layer-3 seeing one link, therefore no ECMP needed in the
routing protocol.  Not zero-touch for sure!

If we assume that (auto)configuring LACP is hard, and want to do in the
routing protocol, I absolutely buy your situation; and your gigabit uplink
while gigabit to NAS makes total sense to me.

It could be that we detect multiple single hops between routers in either the
routing protocol or the HNCP, and we would put both links into the same
broadcast domain. (The architecture says we are supposed to do that)

Given that there might be an arbitrary amount of homenet-unaware L2-switching
gear between A and B,  that seems to imply that one should enable LACP
negotiation in this situation, and it might succeed, or it might not, and if
it fails, then ECMP at layer-3 would make sense.

My impression is that from the point of view of router F (connected behind
your node B), that it would not see any additional adjancy in the routing
TLVs that it sees from B. Maybe the metrics would be better than if there was
only a single link.

What I don't see is a situation where there are actually two equal cost
paths (multiple hops, some not identical) between two end-points within the
home, such that there would be a benefit to ECMP.

In your ABCD+NAS situation, add a wifi link between the NAS and NAS client C.

Assume that in the future OS on C is willing to upgrade to get better
functionality.   My claim is that C will get better benefit from implementing
MPTCP for the connection to the NAS, so that it can go faster when plugged
in, and yet still function if all it has is wifi...  LISP is also an
alternative for the C<-->NAS traffic.  But, if the GbE is plugged in,
(and especially if as you suggest A==B is >1GbE), then I don't see C<-->NAS
traffic ever desiring to use wifi at the same time as wired.

{I have 50Mb/s DSL2 at home now, and in anticipation, I had previously
replaced my 24-100Mb/s "smart" switch with a major-brand 24-port GbE SOHO
managed switch in 2012... Damn thing actually can't keep up with back to back
packets between NFS server and desktop... even at 100Mb/s... Turing QoS *OFF*
reduced my drop-rate from 1 in 2, to 1 in 10... Ridiculous.}

---

From: Dave Taht <[email-masked]>
Date: Wed, 12 Aug 2015 18:05:17 +0200

> use cases are is necessary in order to get something done.

I am extremely busy with make-wifi-fast, but have two cents to add in
here, related to congestion and utilization, and
what I would call staged routing updates (please tell me a better
name), giving similar results to ecmp. instead of converging on one
best route, multiple "best" routes are chosen.

Consider a network with the following topology, all gigE.

A  B C  D
------------  (switch)
     |  |
    E F
    |   |
----------   (switch)
G H I K

E and F are equal cost. In ecmp packets are basically FQ'd across E
and F, resulting in massive packet loss for all flows if either E or F
fails - or weird delays as one becomes overutilized.

If instead, A and B chose E, and C and D chose F, then we get flows
utilizing the full bandwidth available, with no possibility of
re-ordering (except on a route change), but no fq (so we can't get
twice the bandwidth for any given pair of hosts), which seems to me to
be more of the "right thing".

By "staged routes" I kind of mean - make a small route change, then
measure the result, then make more, until the links are as balanced as
possible. There is no single "best route" here, and indeed, the "best
route" for all stations is that they all choose a different E or F.

-- 
Dave Täht
worldwide bufferbloat report:
http://www.dslreports.com/speedtest/results/bufferbloat
And:
What will it take to vastly improve wifi for everyone?
https://plus.google.com/u/0/explore/makewififast

---

From: Mikael Abrahamsson <[email-masked]>
Date: Wed, 12 Aug 2015 18:44:25 +0200 (CEST)

>    incompatible manner or design a new extension)?

To avoid complication, I would be fine with just saying "if two directly 
wired connections of the same speed is established between two routers, do 
ECMP" and don't do ECMP in any other case unless manually configured (I 
have no idea how much the last part makes things complicated).

Basically, trying to get ECMP to work over media with different speed and 
different characteristics sounds very complicated.

> care if they make things worse.)

"caricaturing".

> analysing the Battlemesh results and getting shncpd into shape, and I 
> have a serious teaching load that starts in early September).

Generally ECMP is done when IGP metric ends up identical or when there are 
static routes with the same "metric" pointing to the same place. At least 
in the reality I've been exposed to.

It can also be done over BGP, but this is less common.

I don't know exactly what you're looking for, ECMP isn't that complicated:

https://en.wikipedia.org/wiki/Equal-cost_multi-path_routing

So basically:

A--B
|  |
C--D-E

If metrics of ABDE and ACDE is the same, then program FIB to split L4 
flows between these two paths by means of for instance hashing.

---

From: Juliusz Chroboczek <[email-masked]>
Date: Wed, 12 Aug 2015 19:08:08 +0200

>> Yeah, but if you do it naively, performance will get worse, not
>> better. (The IS-IS community has it easy, they just implement things and
>> don't care if they make things worse.)

> "caricaturing".

Somewhat ;-)

-- Juliusz

---

From: Ted Lemon <[email-masked]>
Date: Thu, 20 Aug 2015 23:25:51 -0400

(This response was delayed by the unplanned obsolescence of my dear departed mail server, but I think it’s still vaguely relevant…)

> and probably not really scalable for a whole country.

My wife and I live in a Passivhaus that we built a couple of years ago.   The Polyisocyanurate insulation on the outside has a foil backer to reflect heat.   It also reflects radio waves.   This means that my garage (a separate building) is totally isolated from the inside of my house, as is the front yard.   This situation is going to become more, not less common as advanced building techniques turn more and more houses and apartments into faraday cages.

So it’s not the case that seamless WiFi handover is solely an issue for community wifi.   It’s an issue for my home network _right now_.

---

From: Ted Lemon <[email-masked]>
Date: Fri, 21 Aug 2015 00:19:27 -0400

On Aug 11, 2015, at 7:25 PM, Alia Atlas <[email-masked]> wrote:
> It sounds to me like using multiple paths (ECMP or otherwise) is something that hasn't been clearly nailed down in the requirements? 

Putting ECMP in the requirements seems like a terrible idea.   I don’t think there’s a need for it, for two reasons.   First, none of the applications you described actually require it.  Yes, you’d get better performance if they had it, so it would be a nice value-added feature.   But it is not a base requirement for the homenet to function.   Second, did you read my description of the typical homenet user’s mental model of what a homenet looks like a week ago?   I think this is a key point: the end user has no idea what ECMP is, what its operational characteristics are, what links it might function over, etc.   So ECMP would have to self-configure, and that includes the sort of stuff Juliusz was talking about—noticing interfering versus non-interfering paths, etc.   This is a research project.   I think it would be great if homenet could do this work at some point in the future, but it is not something that should be part of the base requirements for the homenet, because if it were, it would delay availability of a complete homenet spec by years.

---

From: Alia Atlas <[email-masked]>
Date: Tue, 25 Aug 2015 11:46:10 -0400

Hi Ted,

On Fri, Aug 21, 2015 at 12:19 AM, Ted Lemon <[email-masked]> wrote:

> > It sounds to me like using multiple paths (ECMP or otherwise) is
> something that hasn't been clearly nailed down in the requirements?
>
> Putting ECMP in the requirements seems like a terrible idea.   I don’t
> think there’s a need for it, for two reasons.   First, none of the
> applications you described actually require it.  Yes, you’d get better
> performance if they had it, so it would be a nice value-added feature.
>  But it is not a base requirement for the homenet to function.   Second,
> did you read my description of the typical homenet user’s mental model of
> what a homenet looks like a week ago?   I think this is a key point: the
> end user has no idea what ECMP is, what its operational characteristics
> are, what links it might function over, etc.   So ECMP would have to
> self-configure, and that includes the sort of stuff Juliusz was talking
> about—noticing interfering versus non-interfering paths, etc.   This is a
> research project.   I think it would be great if homenet could do this work
> at some point in the future, but it is not something that should be part of
> the base requirements for the homenet, because if it were, it would delay
> availability of a complete homenet spec by years.
>

ECMP or downstream paths is not a research project; it is common used
technology.  When the traffic streams desired are larger than can fit
across a single path, it becomes critical.

Figuring out how to handle interfering vs. non-interfering paths is, I
think, orthogonal to ECMP.  The multiple links
that might interfere can easily be used for different destinations.  While
interesting, that seems like a problem that can needs
solving already.  Is that piece of the problem a research project?
Figuring out what links interfere?  Is this something that would need
a centralized view of the home network?

>From a user's perspective, use of multiple paths would be transparent -
except that they'd see better performance through
their network.  There can be high-bandwidth demands like data backup or
streaming multiple high-def video.

Regards,
Alia

---

From: Ted Lemon <[email-masked]>
Date: Tue, 25 Aug 2015 12:43:59 -0400

On Aug 25, 2015, at 11:46 AM, Alia Atlas <[email-masked] <mailto:[email-masked]>> wrote:
> ECMP or downstream paths is not a research project; it is common used technology.  When the traffic streams desired are larger than can fit across a single path, it becomes critical.

ECMP in a homenet environment, self-configuring and reliable, is a research project.

> Figuring out how to handle interfering vs. non-interfering paths is, I think, orthogonal to ECMP.  The multiple links
> that might interfere can easily be used for different destinations.  While interesting, that seems like a problem that can needs
> solving already.  Is that piece of the problem a research project?  Figuring out what links interfere?  Is this something that would need

This is not a problem that a typical end user needs solved.   You do not need ECMP for 4k Netflix.   You do not need ECMP to talk to your IoT devices.   You don’t even need ECMP to talk to your homenet file server, in the currently unlikely event that it’s not in the cloud (that is, in a rack in a data center outside the home).   ECMP might be _nice_ for the unlikely in-home file server case, but it is not _necessary_.   If you think it is, it’s probably because you are used to low-performance home routers with bufferbloat, a problem ECMP would not address, and likely would make worse.   We’ve all experienced the home router that, when you try to do a massive file transfer between two devices, suddenly stops routing anything else until the transfer is finished.   This is not a problem that ECMP would fix.

What I consider to be a research project are:

- Figuring out that links interfere
- Figuring out what set of links are candidates for ECMP for a particular pair of endpoints
- Handling unannounced topology changes

To me this is a classic case of premature optimization.   Of course ultimately we’d like this to work.   But it’s not even remotely something that we care about _right now_.   If we ship homenet devices that do not do ECMP, nobody will even notice.

---

From: Alia Atlas <[email-masked]>
Date: Tue, 25 Aug 2015 13:02:23 -0400

Ted,

On Tue, Aug 25, 2015 at 12:43 PM, Ted Lemon <[email-masked]> wrote:

> On Aug 25, 2015, at 11:46 AM, Alia Atlas <[email-masked]> wrote:
>
> This is not a problem that a typical end user needs solved.   You do not
> need ECMP for 4k Netflix.   You do not need ECMP to talk to your IoT
> devices.   You don’t even need ECMP to talk to your homenet file server, in
> the currently unlikely event that it’s not in the cloud (that is, in a rack
> in a data center outside the home).   ECMP might be _nice_ for the unlikely
> in-home file server case, but it is not _necessary_.   If you think it is,
> it’s probably because you are used to low-performance home routers with
> bufferbloat, a problem ECMP would not address, and likely would make worse.
>   We’ve all experienced the home router that, when you try to do a massive
> file transfer between two devices, suddenly stops routing anything else
> until the transfer is finished.   This is not a problem that ECMP would fix.
>

It depends on link speeds in part.  My point was merely that this is a
default in link-state routing. It is self-configuring, of course. I was
asking to understand the requirements.

> - Figuring out that links interfere

Is this merely the trade-off between using multiple links?  I normally
assume that some traffic is carried
across all links.  I am not sure why the concept of using multiple paths is
radically different than the basic
problem or why you feel it explodes the complexity?

> - Figuring out what set of links are candidates for ECMP for a particular

In most routing protocols, this is trivial and done.  I do believe that
there is work to be done for Babel in
this space, if it is needed.

> - Handling unannounced topology changes

?? Isn't that what a routing protocol does - detects and distributes the
topology change?

> To me this is a classic case of premature optimization.   Of course
> ultimately we’d like this to work.   But it’s not even remotely something
> that we care about _right now_.   If we ship homenet devices that do not do
> ECMP, nobody will even notice.
>

Ted, I asked a question about a feature that is considered critical in
every routing environment that I am familiar with.
I find it frustrating that looking ahead to significantly more complex home
networking topologies and link types, which may be
many years out still, is unexceptional, but asking about a feature that
allows better use is described as premature optimization.
I am asking about a routing requirement.

I still am not clear on how link interference is handled for different
destinations.

Regards,
Alia

---

From: Michael Richardson <[email-masked]>
Date: Tue, 25 Aug 2015 13:03:41 -0400

Ted Lemon wrote:
    > week ago? I think this is a key point: the end user has no idea
    > what ECMP is, what its operational characteristics are, what links
    > it might function over, etc. So ECMP would have to self-configure,
    > and that includes the sort of stuff Juliusz was talking
    > about—noticing interfering versus non-interfering paths, etc. This
    > is a research project. I think it would be great if homenet could
    > do this work at some point in the future, but it is not something
    > that should be part of the base requirements for the homenet,
    > because if it were, it would delay availability of a complete
    > homenet spec by years.

    > think, orthogonal to ECMP.

But, Ted's point is that without a way to identify what are the
non-interfering paths are, that is, the non-interfering *WIRELESS* paths,
then ECMP won't be useful except on wired links.

Wired links are already fast enough, and in the cases where they aren't,
and one plugs two GbE cables between two routers, it seems that we don't
need ECMP, just automatic link-aggregation, to make it work.

    > streaming multiple high-def video.

They could be made transparent, but we don't think that they can be
made that way for wifi, which is where the major benefit would be.

---

From: Henning Rogge <[email-masked]>
Date: Tue, 25 Aug 2015 19:17:08 +0200

On Tue, Aug 25, 2015 at 7:02 PM, Alia Atlas <[email-masked]> wrote:
> Ted, I asked a question about a feature that is considered critical in every
> routing environment that I am familiar with.

The options I know are "ignore it" or "include the interference domain
size into the link cost".

Still, using multipath in wireless environments can be tricky... it is
quite easy to mess up even with two paths and get less throughput than
from a single path.

There is also the point that multipath choices tend not to be
isometric... just because the two paths from your local point of view
seem to be good they are not necessarily good from the point of view
of the next hop.

Henning Rogge

---

From: Juliusz Chroboczek <[email-masked]>
Date: Tue, 25 Aug 2015 19:37:25 +0200

> every routing environment that I am familiar with.

I think that we all have different pictures of what a homenet will look
like.  Some of us appear to believe that homenets will predominantly
consist of wired links with a few WiFi access points at the edges, while
others (including myself) think that as soon as we give the hungry masses
the ability to build self-configuring networks that are efficiently routed
at layer 3, people will start building networks where wireless is used for
transit.

This is exciting stuff, Alia, more exciting than some of the contributors
to this list seem to realise.

Now the only goal of ECMP is to improve performance.  If the homenet uses
only wired links for transit, then ECMP is easy to do, whether in IS-IS or
in Babel.  If, however, the homenet is using multiple mutually interfering
radio links for transit, then a naive implementation of ECMP will actually
decrease performance due to interference (cross-link collisions).  Doing
ECMP in the particular case of non-interfering paths (e.g. wired paths)
should be safe, and the radio interference extension to Babel should
already carry all the required information.

Thankfully, ECMP does not have to be a Homenet requirement -- if the ECMP
extension is backwards compatible (and there's no reason why that
shouldn't be the case), then we can simply leave it as an implementation
option, and let the market decide.

I realise you're impatient to see ECMP in Homenet, Alia, but my main
priorities right now are to satisfy the requirements of the IETF
secretariate wrt. the Babel extension drafts (I've missed the IANA
requirements in one of the drafts), getting shncpd into shape for
integration into Debian, and helping with the Babel/Bird implementation --
and teaching starts in two weeks!  Please give me a few months until
things calm down a little.

-- Juliusz

---

From: Alia Atlas <[email-masked]>
Date: Tue, 25 Aug 2015 13:38:24 -0400

Hi Henning,

On Tue, Aug 25, 2015 at 1:17 PM, Henning Rogge <[email-masked]> wrote:

> every

Ok - it makes sense to me about including it in the link cost.

> quite easy to mess up even with two paths and get less throughput than

So the concern is getting worse throughput to the same destination - but for
different destinations, one just hopes the information in the link cost are
sufficient?

> seem to be good they are not necessarily good from the point of view

In a way that can't be captured by link metrics?  I haven't really looked at
the unique characteristics for wireless.  I'm happy to do a bit of
self-education.

Thanks,
Alia

---

From: Henning Rogge <[email-masked]>
Date: Tue, 25 Aug 2015 19:50:23 +0200

> the unique characteristics for wireless.  I'm happy to do a bit of

Imagine a network with three wireless routers (A,B,C)... A is the
uplink, you are C, but both A and C can only see B.

All routers are dualband routers (all have both a 2.4 GHz and a 5 GHz radio).

>From your (C) point of view the multipath-solution is "easy", one path
use 2.4 GHz (C to B to A), the other one uses 5 GHz (C to B to A).

But when your IP packet arrives at B, B doesn't know it is part of a
multipath stream... so forcing both streams to stay on their frequency
is not trivial if you don't want to do source routing.

There is a solution for this easy example (as Juliusz will certainly
be able to tell you), but there are more complicated setups which are
even more difficult.

Multipath on wireless links is easy to mess up, so I would suggest NOT
including it into the feature-set required by Homenet.

Henning Rogge

---

From: Alia Atlas <[email-masked]>
Date: Tue, 25 Aug 2015 13:51:27 -0400

Hi Juliusz

> to this list seem to realise.

Absolutely on the exciting stuff and on the ability to make a real impact.
On the different pictures, I agree totally & that's why I'm poking a little
into
assumptions & requirements.

> things calm down a little.

Actually, I don't have a strong opinion on whether ECMP is needed in
homenet!
That's why I'm asking - but I'm hoping for technical reasons and trying to
understand
why the interference isn't a general problem.  Perhaps it's just because
for ECMP/multi-path,
there is a choice and trade-off to be examined as to the improvement.

I'm sure you are busy!

Regards,
Alia

---

From: Alia Atlas <[email-masked]>
Date: Tue, 25 Aug 2015 14:06:34 -0400

Hi Henning,

> is not trivial if you don't want to do source routing.

Ok - I was assuming that each router would hash and pick a next-hop for the
traffic.  So traffic that came in on a 2.4 GHz could go out the 5 GHz.
With consistent hashing, maybe one could force traffic to specific paths,
but that isn't usual for ECMP.

> including it into the feature-set required by Homenet.

Ok.  Let me poke a bit more.  Is it safe to use parallel wireless links
between
two routers A and B?  Consider that there's a square topology, as below,
where E<->F and
D<->E are both wireless, and links have the same metric.  Can C safely send
traffic
to reach F via both D and E?  Can F safely send traffic to reach C via both
D and E?
How is this case different than if C were split into a C1 and a C2 so that
F-E-C1 and F-D-C2?

   [ C ]---------[ D ]
     |               | w
     |               |
  [ E ] ---w--- [ F ]

Basically, I'm trying to understand the restrictions.

Thanks for your explanations!
Alia

---

From: Henning Rogge <[email-masked]>
Date: Tue, 25 Aug 2015 20:20:05 +0200

> Basically, I'm trying to understand the restrictions.

Could be, but not necessarily... wireless is always a "shared
broadcast" medium... if any of the links between two routers use the
same frequency, they would interfere with each other. Its more a
matter of position and frequencies than interfaces.

Oh, depending on the hardware even a different frequency would not
give you a perfect separation... the filters of wifi cards are often
for the whole band, so that you get some amount of interference
between all 2.4 GHz (or 5 GHz) channels. But that depends on the
hardware and on the antenna.

Henning Rogge

---

From: Mikael Abrahamsson <[email-masked]>
Date: Tue, 25 Aug 2015 20:38:39 +0200 (CEST)

> at layer 3, people will start building networks where wireless is used for

People already do this, until they discover that their shiny new 1 
gigabit/s Internet connection is now not fully usable to them because 
their wifi isn't fast enough and instead of the traditional case where the 
Internet connection is the bottleneck, now their wifi is the bottleneck. 
I've been in discussions with a substantial amount of people with this 
problem. Either they give up and downgrade their Internet connection speed 
(to save money), they make do with what they have and accept what's going 
on, or they start fixing their wifi so it gives better speed, by upgrading 
to newer equipment and/or doing cabling between their APs in case they 
need multiple.

All the solutions to the wifi speed problem I can see usually means to 
make each wifi base-station smaller, either by means of lower power, or 
higher frequencies. In order to connect these together to achieve good 
quality, you're going to need cables. 60GHz won't go through walls. It 
doesn't really go through anything, you need line of sight.

So my previous opinion stands, I think the homenet routing protocol should 
support ECMP on wired links that are between two directly connected 
devices with identical link speeds. I'm fine to leave the radio interface 
ECMP as a future research project.

---

From: Ray Bellis <[email-masked]>
Date: Tue, 25 Aug 2015 19:47:17 +0100

On 25/08/2015 19:38, Mikael Abrahamsson wrote:

> So my previous opinion stands, I think the homenet routing protocol
> should support ECMP on wired links that are between two directly
> connected devices with identical link speeds. I'm fine to leave the
> radio interface ECMP as a future research project.

Serious question, with no hat on:

In that configuration, why specify ECMP rather than L2 link aggregation?

Ray

---

From: Juliusz Chroboczek <[email-masked]>
Date: Tue, 25 Aug 2015 20:51:39 +0200

> devices with identical link speeds.

No problem with that -- but if it does ECMP, it must be smart enough to
avoid pessimising the traffic in the redundant wireless transit link case.

-- Juliusz

---

From: Mikael Abrahamsson <[email-masked]>
Date: Wed, 26 Aug 2015 00:09:47 +0200 (CEST)

> In that configuration, why specify ECMP rather than L2 link aggregation?

So your suggestion is to opportunistically run LACP on all wired 
interfaces?

My biggest concern is interaction between LACP and HNCP and link-local 
addressing on the interface. As soon as LACP is detected, it needs to rip 
out the IP based config, turn these two interfaces into bonded interfaces, 
put the IP config onto either the bond interface, or on a bridge-interface 
and bridge this to the bonded interface. This is a major reconfiguration 
unless one turns all interfaces into bonded interfaces from the get-go.

With ECMP this isn't needed, with different metrics (I do believe babel 
should gain 32bit metrics as discussed in Prague) for different link 
types, it can be made very unlikely that ECMP would happen between high 
speed wired interfaces and a wifi interface or between two substantially 
different interface speeds. I guess code need to handle if two wifi 
interfaces end up with identical metric though so that ECMP isn't used 
there unless specifically configured.

---

From: Dave Taht <[email-masked]>
Date: Wed, 26 Aug 2015 01:07:26 -0700

On Tue, Aug 25, 2015 at 9:43 AM, Ted Lemon <[email-masked]> wrote:
> technology.  When the traffic streams desired are larger than can fit across
> a single path, it becomes critical.
>
>
> solving already.  Is that piece of the problem a research project?  Figuring
> out what links interfere?  Is this something that would need

There is also the quite common powerline to ethernet bridges. I have
seen these everywhere I have been in france - the network comes in at
one edge of the apartment, the tv is at the other.

http://www.cnet.com/topics/networking/best-networking-devices/power-line-adapters/

Given a choice between using wifi as extensively as we do - or using
powerline technology, I'd lean towards powerline - except that it too,
can suck badly, and
in the context of this conversation, ECMP between regular ethernet and
 a bridged, slow, flaky device like this would also suck.

> bufferbloat,

I note that the de-facto standard for qos+aqm+fq to beat bufferbloat,
in openwrt, is fq_codel based, presently. I generally have not had
enough gumption to insert language into the homenet documents with a
MUST for smart queue management...

but it would help uptake in the industry if it were, indeed,
mandatory. As one example of an existing problem, many, many existing
qos + aqm + fq  systems work correctly with ipv4 only, but blow up
when faced with ipv6. (this is a problem in openwrt's default qos
scripts, fixed in the sqm-scripts, and there is a lot of code out
there, like wondershaper, needing to be updated for ipv6)

Another example of that problem is that it would be nice to have a
standard for fetching "what is my uplink/downlink rate" from isps.
upnp has some support for propagating this info, but it is underused
and rarely configured properly.

> a problem ECMP would not address, and likely would make worse.

+10

-- 
Dave Täht
endo is a terrible disease: http://www.gofundme.com/SummerVsEndo

---

From: Juliusz Chroboczek <[email-masked]>
Date: Wed, 26 Aug 2015 12:16:57 +0200

> There is also the quite common powerline to ethernet bridges.

Yeah.  They look just like Ethernet to the host, so short of speaking the
HomePlug AV management protocol, I don't see how to detect them.  I'm
waiting for Barbara to give me some hints.

> I generally have not had enough gumption to insert language into the
> homenet documents with a MUST for smart queue management...
> 
> mandatory.

Please don't, Dave.  I've spent much of my last months trying to show that
the Homenet stack is reasonably easy to implement.  The reason for that is
that the only unusual data plane feature that we require is source-specific
routing -- everything else is just easy control plane stuff that can be
implemented in software even if you're doing your forwarding in hardware.

-- Juliusz

---

From: james woodyatt <[email-masked]>
Date: Wed, 26 Aug 2015 11:17:07 -0700

> and rarely configured properly.

When I’ve seriously tried pulling hard on that rope in the past, the horrible ugliness at the other end was pretty nauseating.

There usually isn’t a nice “uplink/downlink rate” number that retail providers can advertise. In a DOCSIS network, for example, the CMTS is typically not the narrowest queue in the path to/from the default free zone, physical layer asymmetry means that upward and downward limits are located in different positions, and the limits tend to be dynamic and sufficiently variable that using them for configuring AQM parameters in the subscriber gateway isn’t recommended.

—james

---

From: Ted Lemon <[email-masked]>
Date: Wed, 26 Aug 2015 18:19:51 -0400

On Aug 25, 2015, at 1:02 PM, Alia Atlas <[email-masked] <mailto:[email-masked]>> wrote:
> On Tue, Aug 25, 2015 at 12:43 PM, Ted Lemon <[email-masked] <mailto:[email-masked]>> wrote:
> It depends on link speeds in part.  My point was merely that this is a default in link-state routing. It is self-configuring, of course. I was asking to understand the requirements. 

It depends on a lot of things.   Link speeds are certainly one thing.  Link reliability is another.   Whether the router has a good congestion algorithm.   How fast the router is (can it actually receive packets at the link speed?   can it transmit them at the link speed?).
 
> Is this merely the trade-off between using multiple links?  I normally assume that some traffic is carried
> across all links.  I am not sure why the concept of using multiple paths is radically different than the basic

As a practical matter, I wouldn’t actually expect to see multiple paths other than over WiFi, because in order to get multiple paths otherwise, the user has to plug in two cables between the same device, and they aren’t likely to realize that that would give them more bandwidth (and indeed, depending on the bandwidth of the CPU, it might not).   So if there are multiple paths, they might be kind of wacky, likely aren’t intentional, and there’s no reason to anticipate that they will perform well.
 
> In most routing protocols, this is trivial and done.  I do believe that there is work to be done for Babel in

I’m a bit naive about ECMP, but I _think_ that the "cost" in ECMP is just a metric.   Will that metric be able to represent the real state of the link well enough that it can be automatically used?
 
> ?? Isn't that what a routing protocol does - detects and distributes the topology change?

Er, yes, fair point.   My imagined worry, which may be bogus, is that a flow will get put on a link that was accidentally configured, and then broken when the user fixes the "mistake," but I will admit that that’s a relatively minor concern, particularly if it can be recovered before the connection dies.
 
> Ted, I asked a question about a feature that is considered critical in every routing environment that I am familiar with.
> I find it frustrating that looking ahead to significantly more complex home networking topologies and link types, which may be
> many years out still, is unexceptional, but asking about a feature that allows better use is described as premature optimization.

I may be naive about this.   You seem to think that this will work, and several other people have treated it as plausible.   But I just don’t see much hope that it will actually get used, and so if it’s any trouble at all to get it to work reliably, it shouldn’t be a phase 1 requirement.   I do not mean to suggest that it should never be something that homenets do.