Subject: [homenet] sorting out the right ipv6 addr to choose and name in a source specific world

From: Dave Taht <[email-masked]>
Date: Wed, 17 Dec 2014 06:16:16 -0800

I have been wrestling with prefix coloring, where choosing a "best"
prefix would be of use in (for example) reducing the problems induced
by happy eyeballs when more than one ipv6 prefix is present and
several other scenarios.

There are many parts to this - one is in addressing, the other in DNS,
and this is not dnsmasq specific but kind of general.

* gai.conf

Does anyone have a "better" gai.conf file or getaddrinfo implemention
"out there?

This is the closest I've found to what I sort of think I want.

http://biplane.com.au/blog/?p=122

* hints for addressing?

I will start with the complex scenario - I have three primary
connections to the internet (one is hardline, one is 5 hops away on a
source specific gateway to elsewhere, the last LTE), with slaac, dhcp,
and privacy addresses assigned for each, a ula for internal addressing
(with possibly slaac, dhcp, and privacy addresses), and a vpn (also on
a ula). What the heck, let's throw in a hip address, also. And... why
not? A legacy ipv6 tunnel from hurricane, a 2002:address for 6to4....

The ideal scenario is where the host app picks the "best" address for
each connection type, and I am a little vague on what actually
happens. What my backbrain (but not RFC 3484 - is there a later rfc
for what gai.conf should do?) says should happen is that the host
should pick the shortest prefix match between source and destination.
Well, it turns out this breaks down - my tunnel is a 2001::address,
and my native ipv6 connection is a 2600::address, and my destination
(being on the "legacy" internet), is a 2001::address, so things tend
to pick the tunnel rather than the native address. A differentiation
between these is that I have a different MTU for each. Another is how
these addresses are acquired - the main address comes from dhcpv6,
tunnel from a he speciifc script, the source specific gateway comes
from hnet, the lte address is pppoe....

And worse, the "best" address is dynamic and changes every week, and
on reboots, (this just happened to me and I went nuts flushing old
ipv6 addrs everywhere, since I lack hnetd) and caching it or using it
internally is a bad idea when a static ULA or more static tunnel addr
is available,

The LTE connection costs money to use, and is definately "secondary"
in intent, and I can see supplying that via ra as secondary. I can
also see treating almost all the others as "secondary" also, but it
probably would be nice to have the nearly equal cost hnetd supplied
prefix have a chance at being used occasionally.

and, sigh, the ip route command has no ability currently to set
secondary on an address.

* In terms of DNS

spitting all these into a single DNS record strikes me as insane.
However, having more than one visible address per dns entry strikes me
as desirable. Certainly a dhcp supplied address should end up in dns,
and dnsmasq also has a mechanism to put slaac assigned address into
dns.

putting the ULA in the local dns is actually desirable when you have split dns.

Perhaps having a subdomain per address type? dave.lte.home.lan
dave.slaac.home.lan dave.dhcp.home.lan?

I am not offering any solutions here, just sharing a headache. There
are some other basic problems like setting a default src address on a
default route in this scenario that are giving me headaches in the
source specific universe...

* In terms of being a host and selecting the

for example, I think linux presently choses the last assigned static
address as it's default source, and that can be overridden with stuff
like:

ip -6 route add default via 2001:db8::1 dev ethic src 2001:db8::abcd

but it is not done today, and is not always the correct thing. I add a
vpn address last in my case.

What I see happening now is distributing source specific routes to
hosts doesn't work, if you merely have

default from 2001:558:6045:bd:8dcc:5e18:dd54:b31c via
fe80::6eb0:ceff:fe0b:e2ab dev eth0  proto babel  metric 1024
default from 2601:9:3640::/60 via fe80::6eb0:ceff:fe0b:e2ab dev eth0
proto babel  metric 1024

and no broader default, the right thing doesn't happen for source
address selection on the host. You end up with no ipv6 connectivity at
all until you add something like this

default via fe80::6eb0:ceff:fe0b:e2ab dev eth0  metric 1024

or this

default from :: via fe80::4a1:51ff:fea3:1304 dev eth0.2  proto babel
metric 1024 (different machine)

to get something that works.

On Mon, Dec 1, 2014 at 2:17 PM, Simon Kelley <[email-masked]> wrote:
> On 01/12/14 18:49, Michael Gorbach wrote:
>> On Nov 30, 2014, at 11:17 AM, Simon Kelley <[email-masked]>
>> wrote:
>>>
>>> On 29/11/14 19:18, Michael Gorbach wrote:
>>>> Hi All,
>>>>
>>>> I've got a question and potential enhancement request. It looks
>>>> like right now, the (very useful) interface-name feature pulls
>>>> all (global) addresses from the interface. One of my machines
>>>> uses IPv6 privacy extensions (known in Linux as use_tempaddr),
>>>> which means that in addition to link-local and permanent global
>>>> addresses, it has a rotating cast of ~ 5 temporary addresses. I
>>>> suggest that dnsmasq should detect those temporary addresses and
>>>> not return them for queries that would otherwise hit
>>>> interface-name. Returning them as it does now means > 5 AAAA
>>>> records for a single name, which causes repeated confusion due to
>>>> things like SSH warning about an unknown host because it has
>>>> suddenly picked a previously-unknown temporary address to connect
>>>> to. Thoughts?
>>>>
>>>
>>> Sounds like a sensible suggestion. This facility was added before I
>>> was really familiar with IPv6 and all its extra complications. Most
>>> of those 5 temporary addresses will be "deprecated" ie hanging
>>> around for the use of existing connections, but not used for new
>>> ones. They definitely shouldn't appear, but I'm pretty convinced,
>>> unless anyone can come up with a good reason why not, that all
>>> privacy addresses should be elided, without exception.
>>>
>>> I wonder, though, if that's only true for forward (ie AAAA)
>>> lookups. Should a reverse lookup on an old privacy address still
>>> yield the name of the host it belongs to?
>>
>> Thanks, Simon. I’d agree that all the temporary addresses should be
>> skipped in forward resolution. In terms of reverse, I’d say there’s a
>> high amount of value in having at least the current temporary address
>> resolve to the correct host name. Temporary addresses are often
>> preferred for outbound connections, so if we don’t have reverse
>> resolution here then for example SSH is going to complain that it
>> can’t check reverse DNS. There’s probably some value in reverse
>> resolution for deprecated temporary addresses, for example if you
>> wanted to track down some client in your system logs from several
>> days ago, but it’s significantly lower. If that’s a large amount of
>> work, to me it’s something that wouldn’t be top-priority.
>
> Since all the addresses - permanent, current privacy, and deprecated
> privacy - are there now, it's just a question of filtering them, with a
> bit of extra complication to do it differently for forward and reverse
> queries. I think I've convinced myself that this won't bite anyone if we
> make it a simple change-of-behaviour, rather than some new configuration
> option to request this behaviour. I'd feel more confident if others had
> a think about it and convinced themselves too.
>
>
> Cheers,
>
> Simon.
>
>>
>> Yours, ~ M.
>>
>>>
>>>
>>>
>>> Cheers,
>>>
>>> Simon.
>>>
>>>
>>> _______________________________________________ Dnsmasq-discuss
>>> mailing list [email-masked]
>>> <mailto:[email-masked]>
>>> http://lists.thekelleys.org.uk/mailman/listinfo/dnsmasq-discuss
>>> <http://lists.thekelleys.org.uk/mailman/listinfo/dnsmasq-discuss>
>
>
> _______________________________________________
> Dnsmasq-discuss mailing list
> [email-masked]
> http://lists.thekelleys.org.uk/mailman/listinfo/dnsmasq-discuss

-- 
Dave Täht

thttp://www.bufferbloat.net/projects/bloat/wiki/Upcoming_Talks

---

From: Michael Richardson <[email-masked]>
Date: Thu, 18 Dec 2014 10:07:59 -0500

Dave,
my take is that applications, and the entire gai.conf/getaddrinfo() library
is broken.  Applications can neither be updated nor be trusted to know enough
about the system to be able to make a proper decision.

Somewhere, someone was working on a new connect(2) call that took FQDNs
rather than sockaddr's, such that the kernel could take ownership of this
problem. (Of course, actually solving the problem in a kernel is probably the
wrong answer).
What is necessary is some new infrastructure inside the box which becomes
standardized (like sockets API was), with some daemon that thinks about the
best source addresses is, and possibly gets involved with routing protocols.
(I'm told that OSX has a sophisticated state machine that combines DHCP and
1x, and wifi... it sounds like it could be the start of such a thing)

I think that shim6 and mptcp are answers in this equation.

shim6 has, I'm told, deployment issues which make me very very sad.
mptcp, I'm told, is likely to show up in Apple and Google products and
infrastructure, and my idea (and many others) is that you don't always have
to pick the perfect address for the SYN, just one that works, but rather one
can add better addresses as one discovers them.

--
Michael Richardson <[email-masked]>, Sandelman Software Works
 -= IPv6 IoT consulting =-

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 19 Dec 2014 11:06:45 +1300

On 19/12/2014 04:07, Michael Richardson wrote:
> problem. 

I suppose you are thinking of Name Based Sockets:
http://tools.ietf.org/html/draft-ubillos-name-based-sockets

It's dead, as far as I know.

> (Of course, actually solving the problem in a kernel is probably the
> wrong answer).

Like, it cannot get through most firewalls.

> can add better addresses as one discovers them.

But bad luck if you need UDP.

Some form of intelligent probing does seem to be the answer,
but certainly that needs to be generic because we cannot expect
all apps developers to reinvent it. That's one reason we wrote
http://tools.ietf.org/html/draft-naderi-ipv6-probing recently.

   Brian

---

From: Juliusz Chroboczek <[email-masked]>
Date: Thu, 18 Dec 2014 23:22:55 +0100

Shouldn't we reduce the amount of cross-posting at some point?

> Some form of intelligent probing does seem to be the answer,

I'd like to attract your attention to the work that Matthieu Boutier has
been doing on mosh, Keith Winstein's UDP-based ssh replacement:

  http://comments.gmane.org/gmane.network.mosh.devel/749

Boutier's version of mosh builds connections across all source/destination
pairs, and picks the one with lowest RTT.  It's a work in progress --
there are multiple versions, and Matthieu has yet to decide which
implementation he's going to submit for inclusion in mainline mosh.

We hope to write that stuff down when Matthieu has decided which is the
"right" version, but I'm not promising any hard deadlines -- we have a lot
of stuff that we want to write down.

> all apps developers to reinvent it.

Uh-huh.  But there's only one thing that's worse than generalising from
one example -- it's generalising from zero eexamples.

> http://tools.ietf.org/html/draft-naderi-ipv6-probing recently.

I'll have a look, thanks for the pointer.

-- Juliusz

---

From: Juliusz Chroboczek <[email-masked]>
Date: Thu, 18 Dec 2014 23:46:47 +0100

> for what gai.conf should do?)

With ordinary next-hop routing, routing policy is entirely in the
routers -- the end hosts have no control whatsoever on the path taken.
The opposite of source routing, where the hosts have the ultimate control
on routing policy, and the routers merely provide a mechanism [1].

Now source-specific routing is an interesting compromise between the two:
it splits policy decisions between the routers (who distribute a set of
source-specific routes) and the hosts (who pick among one of the available
routes by specifying a given source address).  But even though the amount
of policy control given to hosts is fairly modest, it still requires that
some routing intelligence be implemented within the hosts -- and, as you
justly note, the available mechanisms are insufficient.

I think we're pretty much done with the routing part -- I'm fairly
confident now that we've got a reasonable understanding of source-specific
routing in edge networks [2,3].  On the other hand, we haven't done any
serious work on the host side of things -- if anyone understands how to
distribute source address selection policies in a way that is both fully
automated and flexible enough, they're not saying.  (And I'm pretty sure
that DNS is the wrong place to do that.)

[1] J. Saltzer, D. Reed, D. Clark.  Source-routing for Campus-wide
    Internet Transport.  1980.
    http://groups.csail.mit.edu/ana/Publications/PubPDFs/SourceRouting.html
[2] M. Boutier, J. Chroboczek.  Source-specific routing.  2014.
    https://hal.archives-ouvertes.fr/hal-00947234v3
    (Yes, it's a new version.)
[3] As opposed to core networks, where there's the ugly issue of routing
    table explosion -- please give Matthieu a core network to break)

-- Juliusz

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 19 Dec 2014 13:09:55 +1300

On 18/12/2014 03:16, Dave Taht wrote:

...
> for what gai.conf should do?) 

Not sure if anyone else has responded to that:

Yes, it was obsoleted by RFC6724. It probably won't change anything
material for this thread, but it's very important for other reasons.

btw I strongly recommend the pages like this one:
http://www.rfc-editor.org/info/rfc3484
if you want to get Obsoleted by/Updated by/Errata information
for any RFC.

   Brian

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 19 Dec 2014 13:19:52 +1300

On 19/12/2014 11:22, Juliusz Chroboczek wrote:
> pairs, and picks the one with lowest RTT.  

Sounds interesting. In the ideal world, that would be a pluggable
policy algorithm. Lowest RTT may not always be the best choice.
NAROS* suggested distributing policy from a single source, for
example.

The point about shim6, of course, is that allows you to change
horses in midstream without bothering the transport layer.
It's a real shame we don't know how to deploy it, especially
for homenets where nobody manages the routing policy.

   Brian

* C. Launois, O. Bonaventure, and M. Lobelle. The NAROS approach for IPv6
multihoming with traffic engineering. volume 2811 of Lecture Notes in
Computer
Science, pages 112–121. Springer Berlin Heidelberg, 2003.

> It's a work in progress --

---

From: Jim Gettys <[email-masked]>
Date: Thu, 18 Dec 2014 19:39:03 -0500

> for homenets where nobody manages the routing policy.

​
What were the problems with getting shim6 deployed?

There appear to have been
​a Linux implementation, and if the idea now has merit, that is enough of
the market (which is very responsive) ​
​to get significant deployment, and to do so quickly. (codel/fq_codel went
from concept to shipping code is under 3 months, with wide test deployment
in a year, and now becoming default).  We aren't talking about 5 year
product cycles any more.

As to policy, the home routers themselves give us a place to enable people
to state the policy they want (e.g. only use the LTE upstream if the cheap
broadband service is unavailable...).
                                    - Jim

> homenet mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/homenet
>

---

From: joel jaeggli <[email-masked]>
Date: Thu, 18 Dec 2014 16:47:14 -0800

On 12/18/14 4:39 PM, Jim Gettys wrote:
>
>
> On Thu, Dec 18, 2014 at 7:19 PM, Brian E Carpenter
> <[email-masked] <mailto:[email-masked]>> wrote:
>
need a time machine

https://www.nanog.org/meetings/nanog35/presentations/schiller.pdf

> ​a Linux implementation, and if the idea now has merit, that is enough
> of the market (which is very responsive) ​
> ​to get significant deployment, and to do so quickly. (codel/fq_codel
> went from concept to shipping code is under 3 months, with wide test
> deployment in a year, and now becoming default).  We aren't talking
> about 5 year product cycles any more.
>
> As to policy, the home routers themselves give us a place to enable
> people to state the policy they want (e.g. only use the LTE upstream
> if the cheap broadband service is unavailable...).
>                                     - Jim
>
>
>        Brian
>
>     [email-masked] <mailto:[email-masked]>

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 19 Dec 2014 14:03:52 +1300

> https://www.nanog.org/meetings/nanog35/presentations/schiller.pdf

Well, let's not start that argument again. Shim6 was never intended to
address operator's concerns, so it didn't. The real world problems are
these:

1. Most firewalls drop packets containing the shim6 extension header.

2. When shim6 switches addresses, the packets get a big bigger and
that might reveal a PMTUD problem.

3. Absent SADR in the exit router, shim6 has a high chance of falling
victim to BCP38 filtering.

More details:
H. Naderi & B.E. Carpenter, Putting SHIM6 into Practice,
Australasian Telecommunication Networks and Applications Conference
(ATNAC 2014),
Melbourne (November 2014).
https://www.cs.auckland.ac.nz/~brian/Shim6-ATNAC14-subm.pdf

    Brian

---

From: Juliusz Chroboczek <[email-masked]>
Date: Fri, 19 Dec 2014 02:49:10 +0100

> policy algorithm. Lowest RTT may not always be the best choice.

It is, in our particular context.  That's the nice thing about working at
the application layer -- you are the application, so you have a pretty
good idea of what the desirable properties are.  Mosh is an interactive
shell, and in this particular context it's latency you want to optimise
for.

Once we gain some real-world experience from mosh, we can think about
making something more generic.  But I, at least, don't feel comfortable
about doing that.

-- Juliusz

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 19 Dec 2014 16:38:10 +1300

> shell, and in this particular context it's latency you want to optimise

Are you sure? Suppose I open a file transfer window (which as a matter
of fact I do almost every time I use SSH these days). For large file
transfers
I might want to specify "cheapest" not "fastest".

> making something more generic.  But I, at least, don't feel comfortable

Understood. But all I'm thinking is that you might make that bit a
self-contained
module.

    Brian

---

From: Dave Taht <[email-masked]>
Date: Thu, 18 Dec 2014 21:39:41 -0800

On Thu, Dec 18, 2014 at 7:38 PM, Brian E Carpenter
<[email-masked]> wrote:
> Are you sure?

For mosh, which is a very low bandwidth, highly interactive application,
lowest RTT is the right choice.

> Suppose I open a file transfer window (which as a matter

I would argue that a transfer over LTE (which is the most expensive option),
would generally exhibit higher RTT over any other method.

lowest financial cost does not have a whole lot of meaning otherwise
along the edge. Highest bandwidth or least congested might be other
selectors...

But: in kicking off this thread, what I was mostly thinking about was
making a choice between 9 or more ipv6 addresses to choose from, what
to start with, and when to give up. (and which ones should end up in
DNS)

It seemed to me that I should longest prefix match, except when other
characteristics of the route interfered. As one example it made sense
to choose a native connection over a tunneled one. As another example,
when there was a vpn in play (with it's own address range) to try to
use the vpn rather than any public addresses that existed. And then
there is a choice to try and prefer stabler addresses (infinite
lifetime) over addresses with shorter lifetimes.

And then, in order to have a better choice of multi-path connectivity,
for an app like mosh or something using mptcp, to chose addresses that
are wildly different. I would argue for an app that wants multiple
forms of connectivity to pick 3-5 addresses total (and for mosh, to
ensure it did ipv4 and ipv6).

I tossed hip in there just to make things difficult, as arguably hip
addresses should only be used when trying to contact another device in
the hip address range ( 2001:10::/28 )

> Understood. But all I'm thinking is that you might make that bit a

Well, I think I was seeking a new API here. But writing a select or
poll loop for 9 addresses is not much harder than 2.

-- 
Dave Täht

---

From: Juliusz Chroboczek <[email-masked]>
Date: Fri, 19 Dec 2014 18:31:37 +0100

>> good idea of what the desirable properties are.

> Are you sure?

This particular debate aside, I think we're in agreement about the
underlying issue -- source selection in networks with source-specific
routing is an interesting problem, and one that we haven't explored much
yet.

> But all I'm thinking is that you might make that bit a self-contained
> module.

Medium-term, yes.

-- Juliusz

---

From: Henning Rogge <[email-masked]>
Date: Fri, 19 Dec 2014 18:39:34 +0100

> routing is an interesting problem, and one that we haven't explored much

You might also need to combine the features of the gateway with the
metric(s) of the path to the gateway.

Its not really useful to select a faster gateway if the path towards
the gateway goes over a slow wifi link.

Henning Rogge

---

From: Matthieu Boutier <[email-masked]>
Date: Fri, 19 Dec 2014 23:54:54 +0100

> the gateway goes over a slow wifi link.

I do end-to-end measurements in my mosh implementation, so we should
not have the problem.  The host selects a source address, in fact a
pair (src, dst), depending on the performances of the whole path
determined by this pair.

Matthieu

---

From: Gert Doering <[email-masked]>
Date: Sun, 21 Dec 2014 13:17:29 +0100

Hi,

> pair (src, dst), depending on the performances of the whole path

This is extremely cool (and quite likely one of the reasons why mosh
performs so well under adverse network conditions).

Gert Doering
        -- NetMaster
-- 
have you enabled IPv6 on something today...?

SpaceNet AG                        Vorstand: Sebastian v. Bomhard
Joseph-Dollinger-Bogen 14          Aufsichtsratsvors.: A. Grundner-Culemann
D-80807 Muenchen                   HRB: 136055 (AG Muenchen)
Tel: +49 (0)89/32356-444           USt-IdNr.: DE813185279

---

From: Henning Rogge <[email-masked]>
Date: Sun, 21 Dec 2014 20:09:22 +0100

> pair (src, dst), depending on the performances of the whole path

Does this really scale well?

If every application on every attached computer in the Homenet
measures the end-to-end performance through every gateway... that
could be a LOT of overhead for larger Homenet installations.

Henning Rogge

---

From: Juliusz Chroboczek <[email-masked]>
Date: Sun, 21 Dec 2014 20:44:05 +0100

>> not have the problem.

> Does this really scale well?

How well do we need it to scale?  Three addresses per host?  A dozen?
A hundred?  (Per host, mind you -- not per router.)

The probes Matthieu's code is sending are less than 90 bytes each.  This
means that if the client and the server have three addresses each, each
probing episode consists of a burst of 9 packets totalling less than one
full Ethernet frame.  (18 if you count the replies.)

On the other hand, if we end up having hundreds of addresses on every
host, the strategy will need to be rethought.  Dave is currently playing
with a network where each host has 10 addresses of different kinds, and
he's trying to work out heuristics to limit the number of probes.  I'd
personally prefer to gather some empirical data before we start optimising
things.

-- Juliusz

---

From: Dave Taht <[email-masked]>
Date: Sun, 21 Dec 2014 12:24:44 -0800

> personally prefer to gather some empirical data before we start optimising

That has been a very long, and interesting, private thread, and I should
summarize here but Ive been awake 24 hours hacking on this -

The particular test server I have is an 8 port rangeley router with 36
projected ip
addresses, connected to a test client with 12, which is rather a lot
of connections to
probe and sort through. Amazingly mosh-multipath, does, indeed, work
in this case.

(And mosh is the greatest thing since sliced bread already for a
beleagured sysadmin type that endures
frequent mobility events and periods of disconnectivity!)

...

I have been also hacking on odhcpd and hnetd and a few other end
products of this wg to see how far they are along.  (my testbed plan
was to have 5 source specific gateways working across about 70
routers)

I have had to file multiple bugs on all the dhcpv6 clients, and be
annoyed at comcast for sending ras under a minute intervals,
thoroughly break ubuntu and openwrt,  and work around various other
problems, but overall things are much better than they were last
month, and almost as fast as I file bugs, they get fixed.

see various open and closed bugs on https://github.com/sbyx/hnetd/issues/24
https://github.com/sbyx/odhcp6c/issues/27

and rants about it all and some of the tools I have been using on my g+

https://plus.google.com/u/0/107942175615993706558/posts/CsGyyJspjVK

But everything homenet-ish is looking very promising. I guess I have
to go compile a couple boxes with mptcp support soon.

so far babel has been dealing with failing over default gateways going
up and down and source specific routes through a very complex network
beautifully, even though only 1/3 the network is source specific
enabled. (and updating the parts remaining involves climbing a lot of
trees. in the rain. Preferably with a totally baked version of openwrt
chaos calmer)

-- 
Dave Täht

---

From: Brian E Carpenter <[email-masked]>
Date: Mon, 22 Dec 2014 13:23:37 +1300

> personally prefer to gather some empirical data before we start optimising

A good thought. From the point of view of usefulness, more than two or
three addresses per host seems pretty pointless (that is at any rate
what draft-naderi-ipv6-probing told us). Probe results should probably
be cached for a while and interpreted per-prefix not per-address.

   Brian

---

From: Gert Doering <[email-masked]>
Date: Mon, 22 Dec 2014 12:54:20 +0100

Hi,

On Mon, Dec 22, 2014 at 01:23:37PM +1300, Brian E Carpenter wrote:
> Probe results should probably

I agree in principle, just wonder how big the prefix boundary for 
"per-prefix" should be...  (and I can make cases for "a /32 should be
good enough" or "maybe /48 or even /64 level"...)

Gert Doering
        -- NetMaster

---

From: Markus Stenberg <[email-masked]>
Date: Mon, 22 Dec 2014 12:48:50 +0000

> On 22.12.2014, at 11.54, Gert Doering <[email-masked]> wrote:
> good enough” or "maybe /48 or even /64 level"...)

I would say whatever the prefix length is that is advertised on the link (=SLAAC=/64 typically). Of course, you probably receive savings of only 2x by doing this (for example, temp + perm address). One _could_ argue for larger prefixes if you could somehow intuit that they are from same ISP and use same uplink, but what is preventing ISP from applying different traffic shaping on them if they for some reason give you multiple prefixes?

So for simplicity’s sake, I think that the per-prefix-on-link-granularity is the sane level.

I have been thinking about how to abstract this stuff from applications too; ultimately there are two semi-separate things that need to happen;

- SA selection
- DA selection (+IPv4/IPv6)

and in case of TCP apps, assuming you are willing to skip DA selection (for apps that do not care about it), you could easily enough make SA-choice-only ~HE that would work with every app, by doing simply multiple connections in the background and just using the one that succeeds connecting fastest. This could be patched inside C library (or overridden in the C library using LD_PRELOAD) to get it working on any app without app level changes. Of course bonus points for storing results for awhile so no need to probe on every connect().

For UDP, you seem need app support in any case. 

For more ambitious TCP solutions, such as for multiple DA (or simultaneous multiple SA) you ultimately want mptcp anyway, and failing that, connect2() with DNS name argument + IPv4/IPv6 agnosticity. But changing SA/DA on the fly in TCP is not really an option, so you can do it only on per-connection basis probably which more or less sucks if you have long-lived connections. (Not so big problem in most web browsing I guess.)

Cheers,

-Markus

---

From: Gert Doering <[email-masked]>
Date: Mon, 22 Dec 2014 14:04:58 +0100

Hi,

On Mon, Dec 22, 2014 at 12:48:50PM +0000, Markus Stenberg wrote:
> > good enough??? or "maybe /48 or even /64 level"...)
> 

I was actually looking the other way, destination space - if you know
that for, say, 2001:608::1 the path over ISP A is "better" (for whatever
local metric), everything else inside 2001:608::/32 will have the same
result for the same metric, as it's really a single network in a single
city with identical routing policy for all of the /32.

But this is just this particular ISP, while others might have vastly
different routing policyies for different /48s out of the same /32, like
"Akamai"...

> So for simplicity???s sake, I think that the per-prefix-on-link-granularity is the sane level.

I have the nagging suspicion that "permanent + temp addr" wasn't even on
Brian's radar here (and might not actually be relevant :-) - if you have temp
addresses, you don't connect outbound from the permanent address in the 
same prefix, no?).

> and in case of TCP apps, assuming you are willing to skip DA
> selection (for apps that do not care about it), you could easily
> enough make SA-choice-only ~HE that would work with every app, by
> doing simply multiple connections in the background and just using
> the one that succeeds connecting fastest.

Mmmh, interesting idea, yes.

Gert Doering
        -- NetMaster

---

From: Markus Stenberg <[email-masked]>
Date: Mon, 22 Dec 2014 13:13:27 +0000

> On 22.12.2014, at 13.04, Gert Doering <[email-masked]> wrote:
> “Akamai"...

Yeah, on DA side, I would not venture to aggregate any results _without_ local data to back it up. So what I would do there is basically gather (ideally on system level and not per application) statistics on difference in the relevant measures (rtt, jitter, bandwidth depending on what you care about) towards different addresses, and try to notice where there is significant change and guess meaningful DA prefix lengths for particular SA based on that.

It does not sound too fun either :)

> addresses, you don't connect outbound from the permanent address in the 

Depends on how stupid (=or portable and standard-API-abiding) applications you have. Typical stack (look at API in RFCs for example) does not expose address lifetimes or {permanent,temp} address type to applications, as the stack is ‘just supposed to do the SA’ on behalf of applications. Now that we are discussing either libraries or applications getting smarter, there are potential problems there.. :p

Sure you can get that with OS specific hackery, but the standardized API is woefully inadequate. And IETF is to blame here to some degree, as the IPv6 APIs come mostly from here.

(Yeah, I guess you can recognize temp/perm address by looking at the bits in the app but ugh..)

Cheers,

-Markus

---

From: Juliusz Chroboczek <[email-masked]>
Date: Mon, 22 Dec 2014 14:50:16 +0100

> Probe results should probably [be] interpreted per-prefix not per-address.

Hmm.  Interesting idea.

I can imagine some scenarios where it could break -- Ethernet bridged with
Wifi (OpenWRT style), mesh networks, or simply a congested, bufferbloated
interface.  I'm not sure how common such scenarios are.

(I'm also not sure how common it is for a host to connect to multiple
peers on the same subnet.)

Empirical data, anyone?

-- Juliusz

---

From: Dave Taht <[email-masked]>
Date: Mon, 22 Dec 2014 06:11:34 -0800

>> Probe results should probably [be] interpreted per-prefix not per-address.

+1. Pick one of dhcp, slaac, or privacy within a prefix with the best
lifetime, maybe?

> Wifi (OpenWRT style),

expound?

>mesh networks,

?

>or simply a congested, bufferbloated
> interface.

It depends on how you get to that interface.

> I'm not sure how common such scenarios are.
>

We are in a brave new world where some providers (or their modems) are
handing out prefix delegations with a dazzlingly short lifetime of
15sec, apparently...

https://github.com/sbyx/hnetd/issues/24

-- 
Dave Täht

---

From: Juliusz Chroboczek <[email-masked]>
Date: Mon, 22 Dec 2014 15:18:47 +0100

>> I can imagine some scenarios where it could break -- Ethernet bridged with

If you have two hosts connected to an OpenWRT router, one over Wifi and
one over Ethernet, they could have very different performance
characteristics while being on the same prefix.

>> mesh networks,

If you have two hosts connected at different places to the same mesh
network, they could have very different performance characteristics while
having addresses from the same /64.

-- Juliusz

---

From: Gert Doering <[email-masked]>
Date: Mon, 22 Dec 2014 15:22:53 +0100

Hi,

> having addresses from the same /64.

Indeed, that's the other end of the diversity spectrum (vs. "all hosts in
our /32 have roughly the same network characteristics").

Gert Doering
        -- NetMaster

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 23 Dec 2014 08:07:45 +1300

> our /32 have roughly the same network characteristics").

Yes. But this whole thing is a heuristic anyway; I could imagine
quite sophisticated algorithms, or something fairly simple such
as "cache results for a pair of /64s; if that gives inconsistent
RTTs, cache results for individual pairs of /128s".

   Brian

---

From: Michael Richardson <[email-masked]>
Date: Mon, 22 Dec 2014 19:12:45 -0500

> be cached for a while and interpreted per-prefix not per-address.

While I would agree that 2-3 makes sense for average desktop system, having
at least one IPv6 address per virtual (http,ftp,smtp,..) host makes lots of
sense, and there could be hundreds.
The host system still needs to pick the right address for outgoing connections.

While light-weight containers likely will put context on the many addresses,
in some container implementations the superuser ("dom0") container sees all
addresses anyway.

I've also observed some systems with dozens of privacy addresses; now I
think that many were marked as deprecated, they ought not be considered, but
as long as something has a socket open, they remain. (HTTP 1.1, ssh
connections...)  I think this will be increasingly common.

--
]               Never tell me the odds!                 | ipv6 mesh networks [
]   Michael Richardson, Sandelman Software Works        | network architect  [
]     [email-masked]  http://www.sandelman.ca/        |   ruby on rails    [

---

From: Dave Taht <[email-masked]>
Date: Mon, 22 Dec 2014 16:22:20 -0800

> On 19/12/2014 04:07, Michael Richardson wrote:

I am way behind on my mail (this thread) and will be away for the holidays.

Merry Christmas, everyone, and to all a happy new year!

>> about the system to be able to make a proper decision.

I concur. Also other apis dating from the 70s, like sendmsg,getmsg,
and the new sendmmsg,getmmsg are a real pita to use, but increasingly
of interest to the webrtc, quic, and brave new post-tcp udp-only
world.

> It's dead, as far as I know.

It's not dead, it's resting!

The kernel work is not that out of date (3 years). Why did progress stop?

https://github.com/namebasedsockets

I for one would really like names to get more closely tied to numbers...

>> 1x, and wifi... it sounds like it could be the start of such a thing)

Openwrt had to go to very tightly integrated internal messaging API in
order to get all of the newly mandated dynamicsm in ipv6 sort of taken
care of, and that still isn't quite working right.

Yes, looking over OSX would be a good idea.

> Like, it cannot get through most firewalls.

I would like to test it against modern firewalls.

> But bad luck if you need UDP.

Meh. The world is seemingly moving to userspace networking anyway.

-- 
Dave Täht

---

From: V6ops <[email-masked]>
Date: Sat, 27 Dec 2014 09:52:51 +0000

Sent from my iThing

> On 23 Dec 2014, at 00:22, Dave Taht <[email-masked]> wrote:
> 
Another potential solution would be to follow current practice in router management. I know Homenet does not wish to change hosts, but it may result in the best solution.

We could:
- create one of more additional software loopback interfaces that are always up and used as source and destination for binding for all (legacy) apps.
- a software interface would have one single /64 prefix from each delegated Homenet prefix, reducing the number of potential sources and sinks.
- allow the end nodes to take part in HNCP as multi-homed stub devices that do not forward packets between hardware interfaces.
- use stub mode of Babel to learn outbound routes, and to originate a return route to the software loopback interface.

That might also solve the conundrum of how to handle roaming beween multiple routed wireless interfaces running on multiple Homenet routers, and should also greatly simplify the namespace, as the AAAA record of the device could be invariant and independent of the attachment point to the Homenet.

---

From: Juliusz Chroboczek <[email-masked]>
Date: Sat, 27 Dec 2014 18:02:03 +0100

> - create one of more additional software loopback interfaces that are
>   always up and used as source and destination for binding for all
>   (legacy) apps.
> - a software interface would have one single /64 prefix from each
>   delegated Homenet prefix, reducing the number of potential sources and
>   sinks.
> - allow the end nodes to take part in HNCP as multi-homed stub devices
>   that do not forward packets between hardware interfaces.
> - use stub mode of Babel to learn outbound routes, and to originate
>   a return route to the software loopback interface.

Yes, that's more or less what we're doing with hosts connected to
a AHCP/Babel mesh right now.  That's also what I suggested we do when
people were arguing in favour of TRILL (giggle) in order to have roaming.

That's certainly something I'm planning to experiment with when I have the
time and energy to replace AHCP with HNCP in our Babel mesh.  (Not before
February, I fear.)

> That might also solve the conundrum of how to handle roaming beween
> multiple routed wireless interfaces running on multiple Homenet routers,

Yes.

> and should also greatly simplify the namespace, as the AAAA record of
> the device could be invariant and independent of the attachment point to
> the Homenet.

Almost -- any given IPv6 might go away when an ISP goes down.  So either
you include all of your ISPs in DNS (in which case you need probing) or
you renumber whenever you change addresses (in which case you need to wait
for DNS to propagate).

In short, it looks like something needs to give.  Unless I'm missing
something (in which case please do tell), multihoming requires one of the
following:

  - hosts perform e2e probing (shim6, MPTCP, or at the application layer); or
  - DNS becomes more dynamic and propagates faster than it does now; or
  - you use PI addresses, with the table bloat that this entails.

I'm not very keen on putting extra load on DNS, and PI seems out of the
question, so it looks like we're stuck with probing.  Or am I missing
something?

-- Juliusz