Subject: [homenet] ipv6 routing mind block

From: Tim Coote <[email-masked]>
Date: Fri, 5 Aug 2016 11:17:24 +0100

Hullo

I’m trying to build a small test environment, so that I can understand how homenet works with multiple routers in the context of low-cost IoT deployments to improve on current approaches. Most of the environment is based on constructing a virtualbox based set of hosts/routers with constrained networking between them.  I think that I’m struggling a bit as I’m not sure enough on what should work, and so cannot pinpoint where the issues are.

Here’s my starting topology:

ISP (Sky) <- (a) -> openwrt based router (1) <- (b) -> openwrt based router vm (2) <- (c) -> debian vm (3)

The vms run on a mac.

(1) has minimal configuration change from CHAOS CALMER (15.05.1, r48532), although there is some nonsense to get it to set up (a). It has a globally routeable address on br-lan (I’m not concerned about revealing these addresses in this network as they move whenever I reboot this device.) The physical router is a linksys wrt1200ac. I’ve had to replace the ISP supplied router as it does not support prefix delegation requests from (2). I’d prefer not to mess with this router and treat it as the edge of the world, rather than buy another adsl link.

(2) is intended to be a homenet router, when I can get the basics working. It’s running [DESIGNATED DRIVER (Bleeding Edge, 49876)]. I think that homenet has set up a correct initial configuration, based on grabbing a delegated prefix of length /62 out of the /56 that (1) got from Sky. The router (2) has put this subnet on its eth2 and assigne that interface a /64 ip address from the /62 subnet.

(3) is irrelevant for the simplest test case described below, but it’s in the layout for completeness.

(a) is an adsl link, bridged to ethernet with a vigor modem, (b) is a virtualbox bridge to the wifi device of the mac, (c) is a virtualbox ‘internal network’.

I would expect to be able to ping the ip address (2a02:c7d:1d31:a605::3c/64) on eth2 of (2) from (1). I can see the icmp6 packets arriving at eth1 of (2), but they don’t seem to be forwarded to eth2.

09:48:57.023871 IP6 (hlim 64, next-header ICMPv6 (58) payload length: 64) 2a02:c7d:1d31:a600::1 > 2a02:c7d:1d31:a605::3c: [icmp6 sum ok] ICMP6, echo request, seq 0
09:48:58.151674 IP6 (hlim 64, next-header ICMPv6 (58) payload length: 64) 2a02:c7d:1d31:a600::1 > 2a02:c7d:1d31:a605::3c: [icmp6 sum ok] ICMP6, echo request, seq 1

In contrast, I can see icmp6 replies on eth1 of (2) when I ping the address on eth1 (just to confirm that there’s not a networking problem)

09:52:53.163874 IP6 (hlim 64, next-header ICMPv6 (58) payload length: 64) 2a02:c7d:1d31:a600::1 > 2a02:c7d:1d31:a600::756: [icmp6 sum ok] ICMP6, echo request, seq 6
09:52:53.163918 IP6 (flowlabel 0x5d10f, hlim 64, next-header ICMPv6 (58) payload length: 64) 2a02:c7d:1d31:a600::756 > 2a02:c7d:1d31:a600::1: [icmp6 sum ok] ICMP6, echo reply, seq 6

I’ve checked that /proc/sys/net/ipv6/conf/*/forwarding are all ‘1' on (2), so it should be forwarding correctly.

IPv4 routeing does work as expected.

Have I missed something obvious, or should this work and there’s an issue somewhere in the technology stacks?

tia

Tim
— cut here for details --

Addresses on (1):
<code>
root@OpenWrt:~# ip -6 addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qlen 532
    inet6 fe80::c056:27ff:fe72:49df/64 scope link 
       valid_lft forever preferred_lft forever
6: br-lan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 
    inet6 2a02:c7d:1d31:a600::1/56 scope global noprefixroute dynamic 
       valid_lft 5882sec preferred_lft 2282sec
    inet6 fe80::c256:27ff:fe72:49df/64 scope link 
       valid_lft forever preferred_lft forever
7: wlan1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qlen 1000
    inet6 fe80::225:9cff:fe13:a436/64 scope link 
       valid_lft forever preferred_lft forever
8: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qlen 1000
    inet6 fe80::225:9cff:fe13:a437/64 scope link 
       valid_lft forever preferred_lft forever
11: pppoe-wan: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1492 qlen 3
    inet6 fe80::a5fe:3e79:733b:d408/10 scope link 
       valid_lft forever preferred_lft forever

</code>

(1)’s routeing table looks sane to me:
<code>
root@OpenWrt:~# ip -6 route
default from 2a02:c7d:1d31:a600::/56 via fe80::ea4:2ff:fe5e:b401 dev pppoe-wan  proto static  metric 512 
2a02:c7d:1d31:a600::/64 dev br-lan  proto static  metric 1024 
2a02:c7d:1d31:a600::/62 via fe80::a00:27ff:fed5:7ed4 dev br-lan  proto static  metric 1024 
2a02:c7d:1d31:a604::/62 via fe80::a00:27ff:fed5:7ed4 dev br-lan  proto static  metric 1024 
unreachable 2a02:c7d:1d31:a600::/56 dev lo  proto static  metric 2147483647  error -101
unreachable fd1f:42ab:5676::/48 dev lo  proto static  metric 2147483646  error -101
unreachable fd5c:15d9:112f::/48 dev lo  proto static  metric 2147483646  error -101
unreachable fd6d:4f5b:4b17::/48 dev lo  proto static  metric 2147483646  error -101
unreachable fd7f:a574:755e::/48 dev lo  proto static  metric 2147483646  error -101
unreachable fdae:52a3:59f9::/48 dev lo  proto static  metric 2147483646  error -101
throw fdbc:8204:46d7::/48 dev lo  proto 73  metric 2147483645  error -11
unreachable fdbc:8204:46d7::/48 dev lo  proto static  metric 2147483646  error -101
fe80::/64 dev br-lan  proto kernel  metric 256 
fe80::/64 dev eth0  proto kernel  metric 256 
fe80::/64 dev wlan1  proto kernel  metric 256 
fe80::/64 dev wlan0  proto kernel  metric 256 
fe80::/10 dev pppoe-wan  metric 1 
fe80::/10 dev pppoe-wan  proto kernel  metric 256 

</code>

(2) addresses:
<code>
root@OpenWrt:~# ip -6 addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 state UNKNOWN qlen 1
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UP qlen 1000
    inet6 2a02:c7d:1d31:a600:a00:27ff:fed5:7ed4/64 scope global noprefixroute dynamic 
       valid_lft 4960sec preferred_lft 1360sec
    inet6 2a02:c7d:1d31:a600::756/128 scope global noprefixroute dynamic 
       valid_lft 7031sec preferred_lft 3431sec
    inet6 fe80::a00:27ff:fed5:7ed4/64 scope link 
       valid_lft forever preferred_lft forever
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UNKNOWN qlen 1000
    inet6 2a02:c7d:1d31:a605::3c/64 scope global noprefixroute dynamic 
       valid_lft 7030sec preferred_lft 3430sec
    inet6 fdd0:c07b:2bec:31::3c/64 scope global noprefixroute dynamic 
       valid_lft 596sec preferred_lft 296sec
    inet6 fe80::a00:27ff:fed7:87e7/64 scope link 
       valid_lft forever preferred_lft forever
6: br-lan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UP qlen 1000
    inet6 fe80::a00:27ff:fe65:6cef/64 scope link 
       valid_lft forever preferred_lft forever
</code>

(2)’s routing table:
<code>
root@OpenWrt:~# ip -6 route
default from 2a02:c7d:1d31:a600::756 via fe80::c256:27ff:fe72:49df dev eth1  proto static  metric 512  pref medium
default from 2a02:c7d:1d31:a600::/64 via fe80::c256:27ff:fe72:49df dev eth1  proto static  metric 512  pref medium
default from 2a02:c7d:1d31:a604::/62 via fe80::c256:27ff:fe72:49df dev eth1  metric 1024  pref medium
2a02:c7d:1d31:a600::/56 from 2a02:c7d:1d31:a600::756 via fe80::c256:27ff:fe72:49df dev eth1  proto static  metric 512  pref medium
2a02:c7d:1d31:a600::/56 from 2a02:c7d:1d31:a600::/64 via fe80::c256:27ff:fe72:49df dev eth1  proto static  metric 512  pref medium
2a02:c7d:1d31:a600::/56 from 2a02:c7d:1d31:a604::/62 via fe80::c256:27ff:fe72:49df dev eth1  proto static  metric 512  pref medium
2a02:c7d:1d31:a600::/64 dev eth1  proto static  metric 256  pref medium
2a02:c7d:1d31:a604::/64 dev eth0  proto static  metric 1024  pref medium
2a02:c7d:1d31:a605::/64 dev eth2  proto static  metric 1024  pref medium
throw 2a02:c7d:1d31:a604::/62 dev lo  proto 73  metric 2147483645  error -11 pref medium
unreachable 2a02:c7d:1d31:a604::/62 dev lo  proto static  metric 2147483646  error -113 pref medium
unreachable 2a02:c7d:1d31:a604::/62 dev lo  proto static  metric 2147483647  error -113 pref medium
fdd0:c07b:2bec:4::/64 dev eth0  proto static  metric 1024  pref medium
fdd0:c07b:2bec:31::/64 dev eth2  proto static  metric 1024  pref medium
throw fdd0:c07b:2bec::/48 dev lo  proto 73  metric 2147483645  error -11 pref medium
fe80::/64 dev br-lan  proto kernel  metric 256  pref medium
fe80::/64 dev eth1  proto kernel  metric 256  mtu 1500 pref medium
fe80::/64 dev eth2  proto kernel  metric 256  pref medium

</code>

---

From: Juliusz Chroboczek <[email-masked]>
Date: Fri, 05 Aug 2016 13:05:21 +0200

> I would expect to be able to ping the ip address
> (2a02:c7d:1d31:a605::3c/64) on eth2 of (2) from (1).

On each router, run

    ip6tables -I INPUT -j ACCEPT
    ip6tables -I OUTPUT -j ACCEPT
    ip6tables -I FORWARD -j ACCEPT

and try again.  (This is not a persistent configuration, it will go away
at next boot).

-- Juliusz

---

From: Tim Coote <[email-masked]>
Date: Fri, 5 Aug 2016 12:20:59 +0100

> On 5 Aug 2016, at 12:05, Juliusz Chroboczek <[email-masked]> wrote:
> 
> — Juliusz

Thanks for the suggestion, but no change.

I’ve also tried taking the firewalls down, and looking at the ip6tables -L -v at different points in time to identify dropped or rejected packets - neither DROPped nor REJECTed counts get above 0.

So I do have a sane view of how the process should be working: packet received on (2), eth1 (as evidenced by tcpdump results on (2)) should be forwarded to the ip address on eth2 (which it doesn’t as tcpdump on that interface shows no packets)

Tim

---

From: Tim Coote <[email-masked]>
Date: Fri, 5 Aug 2016 12:33:14 +0100

I did wonder if Tore’s showstopper bug in odhcpd might be relevant (http://bit.ly/2aXCS3e), but I wanted to ensure that my understanding of how the routeing ought to behave was sound before faffing with grabbing/building a new package.

It ought to be possible, I think, to confirm all relevant configuration changes that odhcpd changes by direct interrogation. But I don’t know what specifics to look at, nor what 'good' looks like.

---

From: Juliusz Chroboczek <[email-masked]>
Date: Fri, 05 Aug 2016 13:37:38 +0200

> Thanks for the suggestion, but no change.

It's difficult to debug remotely, since you're not giving the relevant
information.  (Which interfaces are the routers connected over?  How are
they configured?)

Did you install the "ip" OpenWRT package?  (Check with "opkg status ip".)
We've seen silent failures when busybox ip was being used with hnetd.

If that's not the issue, I suggest checking whether the routing table is
sane and whether babeld and the kernel agree on the routing table
(ip route show, killall -USR1 babeld and check /var/log/babeld.log).
I also suggest checking the neighbour cache (ip -6 neigh show).

-- Juliusz

---

From: Rich Brown <[email-masked]>
Date: Fri, 5 Aug 2016 14:31:51 -0400

Slightly OT: I have posted getstats.sh on the Homenet wiki. This is an adaptation of a script that I created as a troubleshooting tool for OpenWrt. The script collects a standard set of info and outputs it to a single file for easy review. You can also pass in additional commands as arguments on the command line. 

I added the commands mentioned in this thread to make it more useful for debugging hnetd problems.

It's at: http://www.homewrt.org/doku.php?id=getstats_script - I would be pleased to include any additional commands that might be helpful.

Rich

> On Aug 5, 2016, at 7:37 AM, Juliusz Chroboczek <[email-masked]> wrote:
> 
> _______________________________________________
> homenet mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/homenet

---

From: Juliusz Chroboczek <[email-masked]>
Date: Sat, 06 Aug 2016 15:56:03 +0200

> Slightly OT: I have posted getstats.sh on the Homenet wiki. This is an
> adaptation of a script that I created as a troubleshooting tool for
> OpenWrt. The script collects a standard set of info and outputs it to
> a single file for easy review.

https://lists.infradead.org/mailman/listinfo/lede-dev

---

From: Rich Brown <[email-masked]>
Date: Sat, 6 Aug 2016 11:17:48 -0400

Hi Juliusz,

I was waiting to see if anyone on the homenet list found the script to be useful before posting to a wider audience.

Once we find a useful version of the script, my recommendation would be to follow the CeroWrt lead and install it into the build (CeroWrt put it at /usr/lib/CeroWrtScripts ...)

Rich

> On Aug 6, 2016, at 9:56 AM, Juliusz Chroboczek <[email-masked]> wrote:
>

---

From: Tim Coote <[email-masked]>
Date: Sun, 7 Aug 2016 11:09:08 +0100

> On 5 Aug 2016, at 12:37, Juliusz Chroboczek <[email-masked]> wrote:
> 
Understood. I tried to simplify the test case. Missed out the nic names, I now realise.

ip was a separate package on both.

I’ve decided that I need to be more confident that the underlying openwrt’s are behaving themselves. I’ve regressed (2) to a new, clean, minimally configured Openwrt DD. Although this seemed to route correctly to start with, rebooting (2) failed to get a response to IA_PD: I think that I am seeing the bug that Tore identified in the CC release of odhcpd.

So I need to fix odhcpd to get a repeatable test with manually configured routes before I go any further on the hnetd front.

To Rich’s point wrt tooling. I’m having to construct similar tools, although I tend to run them using ssh from a separate machine so that I’ve got centralised control of the environment. That also has the benefit that I’ve got a more complete toolkit than busybox et al (diff, git, rake, etc), making it easier to create repeatable tests and compare results.

---

From: Rich Brown <[email-masked]>
Date: Mon, 8 Aug 2016 07:49:14 -0400

> On Aug 7, 2016, at 6:09 AM, Tim Coote <[email-masked]> wrote:
> 

The system you're setting up sounds fascinating. It would be cool to have something written up about it on the wiki after you think it's solid. (After all, we are creating a system that *by definition* will have multiple routers, and it'll be good to have tools for minimizing administrative pain...)

But as we broaden the pool of people who try Homenet, we can't rely on them knowing as much as those reading this list. 

My hope is to have a baseline set of tools incorporated into the build. We can then say, "Just type this, and report the results back..." and have a hope of diagnosing the problem with the first set of results. (I want to avoid multiple interchanges, saying, "What does X say?" then get that result and say, "What does Y say?" That process is unproductive for developers, frustrating for users, and time consuming for all. For troubleshooting, I prefer to have One Script to Rule them All that's easy to use, even if it includes information that might be irrelevant to the problem at hand.)
 
The getstats.sh script could be the first of these. It collects baseline info for any OpenWrt router plus the commands that people mentioned in this "ipv6 routing" thread. (listed below)

- Thoughts on this strategy?
- Thoughts on the list of commands to include in the script? 

Thanks.

Rich

PS I'm still looking for validation of the commands issued by getstats.sh. (http://www.homewrt.org/doku.php?id=getstats_script) They currently are:

cat /etc/banner
date
cat /etc/openwrt_release
uname -a
uptime
top -b | head -n 20
du -sh / ; du -sh /*
ifconfig
logread
dmesg
opkg status ip
opkg status ipset
opkg status hnet-full
ip -6 addr
ip -6 route
ip route show
ip -6 neigh show
[plus any others you include as command line arguments]

---

From: Juliusz Chroboczek <[email-masked]>
Date: Mon, 08 Aug 2016 16:18:48 +0200

> PS I'm still looking for validation of the commands issued by getstats.sh.

All your commands are peeking at the state of the kernel.  You've got
other layers in your software stack, and when debugging you often need to
find out each of the different states:

  - netifd's state can be obtained by running ifstatus on each interface;
  - babeld's state can be obtained by sending it the USR1 signal and
    extracting the results from /var/log/babeld;
  - I don't recall right now how to get hnetd's state, you'll need to ask
    the guys.

-- Juliusz

---

From: Rich Brown <[email-masked]>
Date: Tue, 9 Aug 2016 12:56:23 -0400

> On Aug 8, 2016, at 10:18 AM, Juliusz Chroboczek <[email-masked]> wrote:
> 

Perhaps I could start factoring the scripts into separate topic areas:

get-kernel-stats.sh # the existing script
get-routing-stats.sh  # a script to gather the info suggested above
... etc.

It's still easier to tell someone to invoke those two commands and send the results back than to engage in multiple interchanges over several days.

Thanks.

Rich